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ABSTRACT 


This thesis provides a comparative analysis of ship Operational Test and 
Evaluation (OT&E) in the United States Navy (USN) and the Royal Australian 
Navy (RAN). It also reviews the acceptance and introduction into service 
procedures for warships in both Navies, including the input from OT&E. This study 
analyses USN and RAN Ship OT&E organisation, policy, and procedures, and then 
compares and contrasts the two systems. The study finds that the RAN OT&E 
system, although originally based on USN OT&E philosophy, now differs in the 
importance, interpretation, application and focus of OT&E. It concludes that to 
achieve efficient and effective trials and acceptance of the new higher risk warships 
currently under construction, the RAN OT&E system needs to be revised. A model 
for OT&E in the RAN is proposed based on the principles derived from the USN 
system. The recommendations include the initial conduct of OT&E in land based 


test sites, followed by dedicated “Whole Ship” OT&E for the first of class. 
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EXECUTIVE SUMMARY 


With new frigates and submarines under construction, the Royal Australian 
Navy (RAN) has embarked on its greatest combatant shipbuilding program since the 
Second World War. Although these combatants are being built to proven designs 
and, therefore, are considered relatively low risk, they do present a greater 
operational risk than recent RAN combatant programs. In September 1991, the 
Chief of Naval Staff (CNS) expressed concern that the RAN may not be capable of 
adequately managing the comprehensive and complex procedures necessary to 
accept the ANZAC ships and Collins class submarines into service. This focussed 
attention on the acceptance procedures, and consequently on the Operational Test 
and Evaluation (OT&E), of the new combatants. The United States Navy (USN) is 
the world leader in the development of high technology weapons systems and have 
extensive experience in their acquisition, testing and operation. As the RAN has a 
similar philosophical approach to acquisition and Test and Evaluation (T&E), the 
USN OT&E system forms a credible basis from which to develop an RAN OT&E 
organisation, policies and procedures capable of adequately assessing the 
operational effectiveness and operational suitability of the new combatants against 


the user requirement. 


A. SHIP TEST AND EVALUATION 


The fundamental purpose of T&E in a system’s development and acquisition 
program is to identify the areas of technical risk to be reduced or eliminated. During 
the early phases of development, T&E is conducted to demonstrate the feasibility of 


conceptual approaches, to minimise design risk, to identify design alternatives, to 


compare and analyse trade-offs, and to estimate operational effectiveness and 
suitability. As a system undergoes design and development, the emphasis in testing 
moves gradually from developmental test and evaluation, which is concerned 
chiefly with the attainment of engineering design goals, to operational test and 
evaluation, which focuses on questions of operational effectiveness, suitability and 
supportability. The principal types of T&E pertaining to ships as defined by the USN 


are: 
1. Developmental Test and Evaluation (DT&E) 


DT&E is conducted to assist the engineering design and development 
process and to verify attainment of technical performance specifications and 


objectives. 


2. Production Acceptance Test and Evaluation (PAT&E) 


PAT&E is testing conducted on production items to ensure systems meet 
contract specifications and requirements, usually for contractual acceptance 


purposes. It is a type of DT &E. 


3. Operational Test and Evaluation (OT&E) 


OT&E is conducted to determine a system’s operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 
modifications to meet established OT thresholds, and develop tactics. OT&E has 


three distinguishing characteristics: 


¢ It is conducted in an operationally representative environment. 


¢ It is conducted on production representative equipment using fleet personnel 
for operation and maintenance. 


¢ It is conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 


The focus of DT&E is on a system meeting technical and production 
specifications, while in OT&E, the focus is on assessing the actual functioning of the 


system in the realistic combat environment, against the user requirement. 


B. OT&E ENVIRONMENT 


The U.S. places a high priority on Defence with strategies of world leadership, 
global influence and self sufficiency. To achieve these strategies, the U.S. devotes a 
high percentage of its Federal expenditure to Defence, and places a high priority on 
Research and Development (R&D). The USN need for OT&E stems from its almost 
exclusive reliance on indigenous weapons development and production. It has a 
history of large, risky, developmental programs which push the state of the art in 
weapons technology. To minimise technical and operational risk, the outputs of 
these programs require the assessment of operational effectiveness and suitability 
against the user requirement before being committed to production and subsequent 
introduction into the Fleet. 

Australia’s Defence strategy has progressed from a position of dependence on 
allies, to a positive acceptance of both self-reliance and regional influence. To 
achieve its Defence goals, however, Australia devotes much less of its resources and 
places less priority on R&D than does the U.S. The need for local T&E is recognised 
by the Australian Government and DoD, for both indigenous and overseas systems, 
however, the importance of OT&E in particular, is not specifically addressed in any 
Australian T&E policy document. This lack of recognition of the importance of 
OT&E is perhaps due to the RAN procuring low risk, complete ships in the past 
(e.g., FFG-7 class) with the wealth of USN DT&E and OT&E behind it. 


C. OT&E IN THE ACQUISITON PROCESS 


The U.S. Defence system acquisition life cycle consists of progressive 
development phases separated by major decision milestones, when a program is 
reviewed and authorised to advance to the next stage. The USN OT&E system forms 
an integral part of the U.S. DoD acquisition process. It is an empirical method of 
ensuring a sufficient technical return on acquisition investment and for ensuring that 
a new system is fully capable of meeting the fleet’s needs. Asa result, the U.S. place 
high importance on OT&E, with its conduct being mandated by Congress and 
incorporated in the law of the U.S. OT&E is viewed as being more important as a 
basis for a decision to proceed beyond Low Rate Initial Production, than on the 
introduction into service of the final production item. This is because the “big 
bucks” of most Defence acquisitions are spent during the production phase. OT&E, 
however, has started to play an important role also in the assessment of first of class 
systems for introduction into the USN Fleet. 

The USN and RAN have similar acquisition systems, however, the level of 
importance of T&E, and OT&E in particular, differs greatly. The Australian DoD 
prefers lower risk, proven technologies for the majority of new acquisitions. The 
Australian system, therefore, places a high T&E priority on the post production 
phase, with no mandatory requirement for OT&E prior to a production or purchase 
decision, but requiring OT&E before the first production system is Accepted into 
Naval Service (AINS). As Australia is now buying far less “off the shelf” systems 
from overseas, and systems are increasingly being designed and built in Australia or 
adapted from overseas designs, the operational risk of these systems is increasing. 
These greater risks place emphasis on OT&E leading to a production or purchase 


decision where the majority of a project’s funding is spent. 


D. OT&E ORGANISATION 


The organisation of OT&E within the USN is complex with separate authorities 


performing a number of OT&E functions, as listed in Table 1. 


TABLE 1: USN OT&E FUNCTIONAL AUTHORITIES 


OT&E Function USN Authority 


Policy Maker & Overseer Director, Operational Test & Evaluation 
(DoD authority) 






Sponsor Chief of Naval Operations 


Developing Agency Commander, 
Naval Sea Systems Command 
Operational Tester & Evaluator Commander, 
Operational Test & Evaluation Force 
User Fleet Commanders, 
Ships’ Commanding Officers 
Coordinator Director, Test, Evaluation and Technology 
Requirements 


Significant characteristics of this organisation include the appointment of a 






Director of OT& E (DOT&E) responsible for policy formulation, evaluation and 
oversight of OT&E within the U.S. DoD, and the inclusion of a coordinator for USN 
T&E. The USN also separates the OT&E and DT &E organisation structures. 
Following a number of recent reorganisations, the RAN’s OT&E structure is in 
disarray. There is no clear OT&E, or T&E, policy maker or overseer within the 
Australian DoD or the RAN. The Sponsor is not within the RAN organisation, being 
part of centralised force development and user requirements organisation (HQADF) 
who sponsors major RAN acquisitions, and is responsible for the initial 


determination of T&E requirements. The Developing Agency (ACMAT-N), unlike 


its USN counterpart, does not have a T&E office to support Project Directors. With 
the disestablishment of the RAN Trials and Assessing Unit (RANTAU) in May 
1992, the RAN’s OT&E authority is now the Commander, Test and Evaluation 
(CTE) under the Chief Staff Officer (Engineering) (CSO(E)) in the Maritime 
Command. CTE is currently multifunctional, conducting DI&E and PAT&E in 
addition to OT&E. This results in a lack of differentiation within CTE’s organisation 
between DT&E/PAT&E and OT&E. Also being placed under the in-service 
engineering area of the Maritime Command, he lacks visibility and influence within 
the warfare community and hence credibility in operational matters critical to 
OT&E. CTE’s OT&E responsibilities are overshadowed by his DI&E/PAT&E 
duties. The RAN currently has no single authonty who acts as a T&E focal point 
within Navy Office responsible for coordination of T&E matters, however, a project 
T&E manager does perform this function for his project. 

In summary, the RAN and Australian DoD have reorganised over the past five 
years to create a more effective and efficient user requirements organisation, and to 
better accord with Program Management and Budgeting principles. However, as a 
result, the RAN has no T&E, or OT&E policy maker and overseer, no OT&E 
coordinator within Navy Office, and the RAN’s OT&E authority is buried within the 


in-service engineering management area of the Maritime Command. 


E. OT&E POLICY 

Where USN OTE policy has its basis in the statutes approved by Congress, 
the RAN has no such formal basis and its OT&E policy is embedded in the more 
general T&E policy. Some key aspects of this policy include: 


1. OT&E Policy Documentation 


The USN has a hierarchy of comprehensive OT&E policy documentation 
that provides clear, consistent and non-conflicting policies and guidance to OT&E 
participants, and establishes the disciplined management approach to OT&E taken 
by the USN. In contrast, the current RAN T&E policy documentation provides 
broad T&E policy only and, other than defining OT&E, provides little policy to 
guide its conduct. The RAN has adapted the more detailed USN T&E definitions to 
suit its Own requirements. During this process, however, the RAN lost the 
distinction between the types of T&E, and often confuses OT&E with PAT&E/ 
DT&E. Overall, RAN T&E documentation is not comprehensive, inconsistent and 


fails to provide adequate guidance. 


2. Independence of OT&E Authorities 


Independence of OT&E authorities is the key to the effectiveness of OTRE 
in the U.S. acquisition process. OPTEVFOR, as the independent OT&E agency 
within the USN, reports directly to CNO, while DOT&E provides independent 
oversight and coordination of the military services’ planning and execution of 
OT&E, reporting directly to the Secretary of Defence. The independence of the 
OT&E authorities is designed to ensure impartiality, and honest and open reporting, 
subject to the minimum of political interference. 

Although major acquisition decisions within the RAN are less based on the 
results of OT&E, the RAN also maintains the policy that OT&E is to be conducted 
by an authority independent of the development and production agencies. CTE, on 
the staff of the Maritime Commander, is independent from the development and 
production agencies, but not from the end user, as in the USN. The implications of 
this lack of independence is that the end user may have his own aims and objectives 


which conflict with the total impartiality of OT&E conduct and reporting. 


3. Foreign Weapons Evaluation 


The USN has a foreign weapons evaluation program which is designed to 
support the evaluation of foreign weapons systems, equipment or technology in 
terms of its potential to meet specific U.S. military requirements. The primary 
objective of the program is to reduce the costs of research and development, while 
leading to the acquisition of foreign equipment for U.S. use. Despite the most recent 
Australian Government “White Paper” on Defence Policy requiring the need to be 
able to determine the performance in the Australian environment of equipment of 
both overseas and local origin, no DoD or RAN T&E policy specifically addresses 


the evaluation of foreign systems. 


4. Land Based Test Sites (LBTS) 


Although used primarily for the purpose of conducting DT &E, the USN has 
demonstrated that early OT&E on LBTS can give an estimation of potential 
operational effectiveness and suitability, and hence identify potential operational 
problems early and minimise operational risk. RAN guidance for OT&E using 
LBTS includes a statement that system centres and simulators will be employed for 
early stages of OT&E if available, and that OT-2 may be conducted at a land-based 
test site. The use of LBTS for OT&E during the ANZAC frigate and COLLINS 
submarine projects would appear to offer advantages to the RAN. However OT-2 in 
support of a production decision was not conducted for either the ANZAC frigate 
and Collins submarine project. Although LBTS are being developed for both 
projects, they were not available before production contracts for all the required 
combatants were signed. The use of these LBTS for early OT-3 also was not planned 
in the original project schedules, and so the conduct of OT&E is now subject to them 
being used on a non-interfering basis to the contractor. LBTS should be used for 


OT&E in the RAN to give an estimation of potential operational effectiveness and 


suitability, and hence identify potential operational problems early and minimise 
operational risk. This should be conducted as the first phase of OT-3 during a 
dedicated period following the completion of DT&E on the LBTS. 


5. Clear Delineation between T&E Types 


The USN achieves a clear delineation between the different types of T&E. 
OPTEVFOR conduct only OT&E, and OT&E within the USN is conducted solely 
by OPTEVFOR. Thus, there appears to be good understanding in the USN of what 
is, and what isn’t, OT&E. USN policy also states that combined DT &E/PAT&E and 
OT&E testing should only be considered when there are time and cost savings. In 
addition, this combined approach must not compromise either the developmental or 
operational test objectives, and separate evaluations and reports will be prepared by 
DT&E and OT&E testers. The conduct of combined and concurrent DT&E, 
(PAT &E) and OT&E is not addressed in RAN T&E policy, however, it is implied as 
being almost a requirement. The RAN does not clearly differentiate between the 
types of T&E, which leads to DT&E and OT&E often being conducted over the 
same period, by the same test team. Although the RAN recognises, by definition, the 
difference in objectives and methodology between the types of T&E, no limitations 
or guidance as to the possible hazards of this combined testing approach are 


addressed. 


F. SHIP ACCEPTANCE 


The purpose of ship acceptance in the USN is to ensure delivery to the Fleet of 
complete ships, free from both contractor and government responsible deficiencies. 
Independent verification of readiness of ships for acceptance is the responsibility of 
the President, Board of Inspection and Survey (PREINSURV) who conducts 


Acceptance Trials and Final Contract Trials for each individual new ship. Ship 


acceptance has traditionally been based only on the successful completion of 
PAT&E and material inspections that, for the first of class, culminated in approval 
for Fleet Introduction. However, with the treatment of ships as a complete weapons 
system, as for the DDG-51, Fleet Introduction is now based also on OT&E results. 

In the RAN, Delivery (contractual acceptance) of the ship to the Government is 
a contractual matter and is managed by the Project Director. This is followed by 
RAN trials to assess the ship for Acceptance into Naval Service (AINS). CTE’s 
responsibility is to conduct the trials and report the results. An Acceptance Board 
(AB) is established to provide an assessment of the ship and submit 
recommendations upon which CNS can base his acceptance decision. The President, 
Vice-President and Board members serve on the AB part time, and the sole function 
of the AB is to advise CNS on acceptance matters. It does not conduct tests, trials or 
inspections, and is essentially a board of review. Similar to the USN, the RAN also 
places emphasis on PAT&E leading to AINS, although OT-3 has been included in 
the trials program. However, the trials conducted as OT-3 were really PAT&E / 
DT&E with minimal true OT&E content. For the first of class, OT-4 is conducted 
following AINS. It comprises a series of trials and evaluations over a 12 to 24 month 
period to assess operational effectiveness and operational suitability leading to 
Operational Acceptance. 

The AB, with its broader experience and knowledge than CTE, is generally 
agreed within the RAN as being successful as a board of review of the planning and 
results of T&E for individual projects, and recommending the first of class for 
AINS. As an ad-hoc organisation consisting of various fanenonel specialists 
brought together part time, however, the AB suffers from a number of problems. It 
requires guidance as to its role and functions, T&E philosophy and the procedures 
of the acceptance process. Although the AB attempts to obtain sufficient guidance 


from the T&E documentation, it has found this documentation to be unclear and 
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conflicting. With only DT&E/PAT&E and the minimal operational aspects of the 
OT-3 testing conducted by the RAN, the AB has tended to focus on materiel 
deficiencies, rather than the performance of the first of class against the user 


requirement. 


G. OT&E PROCEDURES 


1. Level and Extent of OT&E 


The determination of the level and extent of OT&E required is a difficult 
question. The fundamental cornerstone of OT&E is the user requirement, generated 
through a series of documents built into the acquisition process. A requirement can 
be traced from the capability need through the refining requirements documentation 
to a specific objective in the TEMP, to the test plan, and ultimately to the final OT&E 
report. This linkage from capability need to operational test program defines the 
scope of testing necessary to evaluate the final product against the current user 


requirement. 


2. Assessing Operational Effectiveness 


Operational effectiveness is the capability of a system to perform its 
intended function effectively over the expected range of operational circumstances, 
in the expected environment, and in the face of the expected threat, including 
countermeasures where appropriate. The USN examines each Critical Operational 
Issue (COI) and decides what needs to be known to enable each issue to be assessed. 
It ensures that the appropriate environments, threats, etc. are included and that 
suf ficient data will be generated to address the COI and objectives. It focuses on 
achieving statistical relevance and making the tests as objective as possible. The 
broad scope of many operational requirements, however, makes their testing and 


subsequent assessment rather subjective, making the determination of meaningful 


1] 


and assessable quantitative Measures of Effectiveness (MOEs) difficult. In addition, 
a lack of resources may reduce an objective, quantitative, statistically relevant test 
schedule to a more subjective, qualitative assessment. These situations require 
expertise and judgement, and compromise between the authorities involved, to 
make the tests as objective as possible. 

The RAN has found it difficult to define exactly what is required to assess 
operational effectiveness. The RAN recognises that the user requirement is the 
bench mark for determining the degree to which a system is effective, and also that 
operational effectiveness is best assessed by a performance demonstration by 
normal operating personnel in the normal or given environment. Assessing 
operational effectiveness has involved analysing each COI and then employing 
modified Ship Qualification Trial or Fleet Exercise Program techniques to evaluate 
them. The RAN has very little guidance on the assessment of operational 


effectiveness. 


3. Assessing Operational Suitability 


Operational suitability is the capability of a system, when operated and 
maintained by typical fleet personnel in the expected numbers and of the expected 
experience level, to be reliable, maintainable, operationally available, logistically 
supportable when deployed, compatible, interoperable and safe. The assessment of 
operational suitability in the USN consists of an Integrated Logistic Support (ILS) 
Plan review, followed by suitability testing based on the expected reliability, degree 
of confidence, thresholds, etc., required to be achieved. OPTEVFOR has analysts 
who design suitability tests and determine measures of suitability, and also evaluate 
and analyse the adequacy of logistic supportability. OPTEVFOR also relies on 
operational personnel to use their experience and knowledge of the system to 


identify inadequate logistic support. 


APs 


Before 1987, the assessment of operational suitability, and ILS in 
particular, was rather subjective and often controversial in the RAN. A new 
approach was taken in 1987 to make the assessment of ILS more objective. The 
RAN is currently developing a routine in-service RM&A data collection and 
analysis system, however, until that is operational, the data is collected and analysed 
on an ‘as required’ basis by CTE. 

In summary, the principles of operational suitability assessment are 
common between the USN and RAN. Although assessments by the RAN have 
become more subjective over recent years, the assessment of RM&A is still in its 
infancy. The RAN requires experienced personnel to design suitability tests and 


analyse results. 


4. OTS&E Planning 


The RAN has embraced the U.S. TEMP concept as the single executive 
document for the management of T&E for major acquisitions. Although required 
early in a project, recent RAN TEMPs have not been raised until after project 
funding has been approved. TEMPS need to be approved earlier to enable inclusion 
of post delivery PAT&E/DT&E and OT-3 in the Project budget, and OT-4 in the 
Maritime Commander’s budget. The authority for DCNS to release the TEMP was 
based on his role as Project Sponsor. With HQADF now assuming this role, the 
DCNS function in TEMP development is unclear. The TEMP approval process 
requires review. 

The USN T&E Coordinator responsibilities include the chairmanship of the 
Test and Evaluation Coordination Group (TECG) for each major program. Some of 
the functions of a TECG are the early definition of terms, measures of effectiveness, 
and the acceptability criteria. In the RAN, a T&E Planning Group (TEPG) is formed 


prior to TEMP development to analyse T&E requirements and estimate the 
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resources required. The TEPG is similar to the USN TECG with one significant 
difference. The USN TECG is chaired by the USN T&E Coordinator under the 
CNO, whereas the RAN TEPG is chaired by each individual Project Director 
through his T&E manager. Despite their best intentions, Project Directors are 
essentially driven by cost and schedule considerations, not the overall T&E 
adequacy of their project. To be truly objective, each TEPG should be chaired by the 


authority responsible for ensuring the adequacy of the RAN’s overall T&E program. 


5. RAN OT&E Reporting 


Reports provide the OT&E authority's conclusions regarding a system’s 
operational effectiveness and suitability, and his recommendations regarding the 
systems future. The USN requires OT&E reports to be impartial, complete, 
thorough and be reported solely by the OT&E authority. 

Within the RAN, the OT&E Authority coordinates the issue of trials 
reports, however, they are issued on completion of each segment of OT&E e.g., o/c 
ASW section of OT-3 etc., usually in addition to a report at the end of the OT phase. 
Quicklook reports are also routinely sent on completion of each week’s testing. The 
ship under test also provides a report, however, it is usually forwarded to its 
operational authority and the Project, in addition to the OT&E Authority. This early 
dissemination of OT&E results, before a full analysis is complete and the 
implications assessed, can lead to other authorities taking hasty action on 
incomplete information, and can prejudice the OT&E authority’s final conclusions 
and recommendations. OT&E results should indicate system deficiencies against 
user requirements rather than equipment defects, so the results need to be fully 


analysed and the implications assessed before being reported to a wider audience. 
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6. Whole Ship OT&E 
The OT-III conducted on DDG-51 was the first time the USN has 


conducted OT&E on a whole ship in a multi-battle situation, in a free play 
environment. The whole ship was viewed as an integrated weapons system, rather 
than as a platform for weapons systems, and OT-III was conducted in a dedicated 
period following Ship Qualification Trials (SQT) and Final Contract Trials. 

Within the RAN, the current OT&E to support an AINS decision (OT-3) is 
little more than an extended SQT period where DT &E/PAT&E is conducted with 
some operational assessment. It is not considered to be adequate to assess the first 
of a new ship class against the user requirement. To assess the operational 
effectiveness and suitability of a ship against the user requirement with the degree 
of confidence required for the Acceptance Board to support an AINS decision, a 


dedicated, free play, scenario based, OT-3 period is required. 


a. OT&E Training 


OT&E is a specialised discipline, with its own philosophy and 
methodology, and so requires a specialist approach with knowledge and experience 
to make it effective. COMOPTEVFOR conducts a three day Operational Test 
Directors’ (OTD) course covering the major areas of OT&E. They also run irregular 
segment courses which provide overviews or updates on OT&E subjects e.g., 
analysis, test plan development and threat updates. 

The RAN has no training courses on OT&E, or on T&E in general. A 
number of officers have completed the USN OTD course in recent years which has 
improved the knowledge and understanding of OT&E within CTE’s organisation 
considerably. These courses have been arranged on an ad-hoc basis through 


overseas Visit submissions rather than as pre-requisite courses for particular billets. 


iy 


To be effective, OT&E requires experienced and knowledgable 
personnel with high professional credibility within their field of expertise. To 
achieve this requires selection of suitable personnel, adequate training and good 
guidance documentation, and preferably a career path where serving in the OT&E 


Authority is seen as career progression by such capable personnel. 


H. CONCLUSION 
The USN OT&E system has, by necessity, developed into a well organised, 


well documented and effective, if complex, system. The RAN system, on the other 
hand, suffers from confusing T&E policy documentation, a weak OT&E 
organisation structure and a general lack of OT&E knowledge and appreciation. The 
determination of OT&E required has traditionally not been achieved early enough 
in the life of a project, and so project funding has not included the provision for 
OT&E. The RAN OT&E system, although originally based on USN OT&E 
philosophy, now differs in the importance, interpretation, application and focus of 
OT&E. As a result of organisation changes, it now lacks the ability to effectively 
manage overall T&E, let alone OT&E, at a time when the operational risk of ship 
projects is increasing. To achieve efficient and effective trials and acceptance of the 
new higher risk warships currently under construction, the RAN OT&E system 


needs to be revised. 


I A RECOMMENDED MODEL FOR OT&E IN THE RAN 

The OT&E model recommended for the RAN uses principles derived from the 
USN system. By taking consideration of resource limitations, and the characteristics 
and culture of the RAN, it establishes an effective OT&E organisation which is 
important to the testing, evaluation and subsequent acceptance decision of the new 


combatants. 
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1. T&E Definitions 


The basis for this OT&E model are clear definitions of the types and phases 
of T&E. The current Australian definitions are amended and phases of T&E are 
refined to better suit the RAN environment. The following definitions of types and 
phases of T&E are proposed for the RAN system: 


a. Developmental Test and Evaluation (DT&E) 


The DT&E definition remains unchanged as follows: 


DT&E is conducted to assist the engineering design and development process, 


and to verify attainment of technical performance specifications and 
objectives. 


DT&E consists of the three phases listed in Table 2. 


TABLE 2: DI&E PHASES 


DT-3 


Demonstration that production meets required technical 
characteristics or establish standards for first of class. 
PAT&E is a form of DT-3. 



















b. Production Acceptance Test and Evaluation (PAT&E) 


The definition of PAT&E is revised to recognise that it is conducted not 
only during the contract period, but also during subsequent RAN testing: 


PAT&E is conducted on production items to ensure systems meet technical 
specifications and requirements, and is a type of DT&E. 
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PAT&E consists of the seven phases listed in Table 3. 


TABLE 3: PAT&E PHASES 













c. Operational Test and Evaluation (OT&E) 


The definition of OT&E is revised along USN lines, so that it more 


accurately describes the purpose and nature of OT&E. 


OT&E is conducted to determine a system's operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 
modifications to meet established OT thresholds, and develop tactics. OT&E 
has three distinguishing characteristics: 


It is conducted in an operationally representative environment. 


It is conducted on production representative equipment using fleet personnel 
for operation and maintenance. 


It is conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 
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OT&E consists of the five phases listed in Table 4. 


TABLE 4: OT&E PHASES 


OT&E Proposed RAN a eeepc ee scription 


forme | Operational assessment of the development proposal to support Full _| assessment of the development proposal to support Full 
Scale Engineering Development approval. 














| aa Demonstration of achievement of program requirements for operational 


| a and suitability to support a production or purchase decision. 






Demonstration of achievement of program requirements for operational 
effectiveness and operational suitability on production of ship/aircraft/ 
system to support Acceptance into Naval Service. Includes limited 
reliability, maintainability, availability and logistic supportability 
assessments. 





Demonstration of achievement of program requirements for operational 
effectiveness and operational suitability on production of ship/aircraft/ 
system in a multi-force, multi-threat environment to support Operational 
Acceptance. Includes detailed reliability, maintainability, availability and 
logistic supportability assessments. 






In-service OT&E, which could include new applications, new tactics, 
revised threat, etc. 


2. T&E in the Acquisition Process 


To ensure the progressive assessment of operational effectiveness and 
operational suitability during the acquisition process, OT&E is scheduled towards 
the end of each acquisition phase. The proposed T&E schedule, noting the 
distinction between DT &E/PAT&E and OT&E, is shown in Figure 1. No change is 
suggested to the current relationship between OT&E phases and acquisition 
milestones, however, the figure separates DI&E/PAT&E from OT&E to clearly 


differentiate these types of testing. 


We 


FULL SCALE 
PROGRAM ENGINEERING PRODUCTION PRELIM 
INITIATION DEV ELOPMENT APPROVAL 

APPROVAL 


PHASE 1 PHASE 2 PHASE 3 PHASE § 


DEMO NST RATION FULL SCALE 
CONCEPTUA AND ENGINEERING PRODUCTION INSERVICE 
EVALUATION DEVELOPMENT 
PAT&E | DT-3/ . 
(TT 0-5) |PAT &E: 
OT&E 
OT-t OT-2 OT-3 OT-4 OT-5 
aTT (ITT 8} 


DT&E/PAT&E 





Figure 1: T&E in the Acquisition Process 


a. Total Ship Test Program 


The Total Ship Test Program (TSTP) concept, modified from the U.S. 
system, is now well established in RAN shipbuilding projects. The proposed T&E 
phases can be incorporated into an amended TSTP. However, instead of being 
grouped together as in the current TSTP, the DT&E and OT&E events are now 
shown as separate categories of testing. Because these separate DT&E, PAT&E and 
OT&E events are integrated within the overall TSTP, these trials are termed 
collectively as “Integrated Tests and Trials’, which is an extension of the current 


RAN application of this term. Table 5 shows the proposed revised categories of ITT. 
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TABLE 5: CATEGORIES OF ITT 


me ies 
a 
a 
A 
Cases Ying 
Sn 


Note: * Only performed on first of class 












3. OT&E Organisation 
The current OT&E organisation structure is modified to enable effective 
OT&E to support the acquisition process. It provides for authorities responsible for 
the management, policy making and oversight, specification and conduct of OT&E. 
This proposed reorganisation closely achieves the aim of DT&E/OT&E separation, 


and is shown in Figure 2. 


Zak 


MINISTER 
FOR 
DE FE NCE 
| secoer | 
DGFDSEA) 


/ 
pom! 
DCNS 7 {| MARITIME 
/ | POMMANDER 
/ | 
PROJECT arr 
spin = 
ae 
COMMANDER] | INSERVICE 
OT& E T&E 


Figure 2: Proposed RAN OT&E Organisation 





Ideally, the head policy maker and overseer of OT&E should be within 
HQADF or DoD, along the lines of the U.S. system. A shorter term, RAN-only 
solution would be to include this function within the revised Director General Naval 
Policy and Warfare (DGNPW) organisation under the Deputy Chief of Naval Staff 
(DCNS). The Sponsor of major naval acquisitions is the Director General Force 
Development (Sea) (DGFD(Sea)) within HQADF. To be able to determine the 
appropriate nature and extent of OT&E required, the Sponsor needs access to OT&E 
knowledge and experience. A close relationship needs to be maintained by 
DGFD(Sea) with the OT&E policy maker and the OT&E authority (Note dotted 
line). To assist all RAN projects with T&E matters, an authority is required within 
the Materiel Division, under the Assistant Chief of Naval Staff - Materiel) ACMAT- 


N), whose responsibilities include assisting Project Directors in complying with 
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policies, incorporating lessons learned from previous projects and reviewing the 
T&E aspects of projects at major milestones. Although primarily a DT&E policy 
authority, this office would also have an input into the OT&E process by reviewing 
the TEMP, and by advising projects how to prepare for OT&E. The Director General 
Naval Engineering Requirements (DGNER) is currently responsible for the design, 
development, acceptance into service and through life support of ships. This 
engineering policy directorate appears to be well suited to the DT&E policy and 
review function. 

The current OT&E Tester and Evaluator, CTE, within the Maritime 
Command is multifunctional, performing both DT&E/PAT&E and OT&E. To 
achieve the advantages of DI&E and OT&E separation, a revised organisation is 
proposed. All CTE’s current DT&E/PAT&E responsibilities, and those already the 
responsibility of CSO(E), would remain with an in-service trials team under 
CSO(E). CTE’s OT&E duties would be removed and managed by a separate group, 
Commander, OT&E (COT&E), working under an operational authority. As OT&E 
is the assessment of operational effectiveness and suitability, COT&E should be 
responsible to COMFLOT. Headed by a Warfare branch officer, COT&E would 
manage the post delivery trials period for all ships from the MHQ perspective. He 
would still manage the SQT, which, although not true OT&E, still has some 
Operational input which could not be met by the in-service trials team under 
S50(E). 

This proposed reorganisation closely achieves the aim of DI&E/OT&E 
separation with some concessions necessary to limit manpower requirements. Also, 
by including OT&E managed by a Warfare Officer under COMFLOT, OT&E could 


achieve visibility, credibility and influence within the RAN. 


2) 


4. RAN OT&E Policy 


a. Policy Documentation 


To correct the current inconsistent and out of date T&E documentation, 
a major rewrite, rather than just updating is required. This rewrite could be the 
vehicle by which the T&E system within the RAN is overhauled, as before the 
documentation is revised, T&E definitions and terminology need to agreed within 
the RAN and DoD. Then the OT&E and DT &E organisations within the RAN need 


to be resolved. and the appropriate policies decided. 


b. Independence of OT&E Agency 


Given the proposed COT&E is organisationally independent from the 
developing and production agencies, his procedural independence should also be 
maintained to ensure impartiality and validity of testing. When OT&E is conducted, 
it should not be influenced by the Project or Design Approval Authority (DAA). No 
Project or DAA personnel should be present during testing. A final report only 
should be issued by the Maritime Commander, with no progress or interim report 
(unless the testing is unusually lengthy), to allow full analysis and evaluation of the 
test results. Similarly, to ensure test validity and impartiality, contractors should not 


be present during OT&E. 


5. T&E Planning 
Test and Evaluation Master Plans (TEMPs) would be approved early to 
enable inclusion of post delivery PAT&E/DT&E and OT-3 in the Project budget, and 
OT-4 in the Maritime Commander’s budget. To be truly objective, each TEPG 
would be chaired by the authority responsible for ensuring the adequacy of the 
RAN’s overall T&E program. The proposed RAN OT&E policy maker and 
overseer, DGNPW, may best be suited to the chairmanship of each TEPG. 


24 


6. Land Based Test Sites 


OT&E would be conducted during a dedicated period in the LBTS 
following successful DT&E, to gain an estimate of the potential operational 
effectiveness and suitability of the system. This period would be planned early so as 


to be included in Project schedules and funds. 


7. Whole Ship Testing 


To assess the operational effectiveness and suitability of a ship against the 
user requirement with the degree of confidence required for the Acceptance Board 
to support an AINS decision, a dedicated, free play, scenario based, OT-3 period is 
proposed. This OT-3 would be conducted after the ship has completed the SQT, 
Workup and Operational Readiness Evaluation to ensure that the testing is 
conducted on a worked up and materially proven ship. OT-3 would be conducted of 


the overall ship as a complete weapons system. 


8. RAN Ship Acceptance 


The AB would be convened for the first of class only, providing the 
specialist expertise to interpret requirements and determine the level of acceptability 
required for AINS. The AB consists of Naval Officers drawn part-time from their 
normal jobs, and so the cost appears to be minimal. The AB, however, has a high 
opportunity cost, which is the value of the next best alternative on which the 
members could be working instead of serving on the Board. Once the first of class 
is accepted the Board’s task is complete. The interpretations and levels of 
acceptability established by the AB then guide the Trials Authority in recommending 
acceptance of each of the remaining ships. With a more comprehensive OT-3 period 
proposed to be conducted, the AB members would more effectively use their 


knowledge and experience in assessing the results of OT&E against the user 
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requirement rather than on workmanship and materiel deficiencies. The AB should 


remain as a board of review. 
9. RAN OT&E Procedures 


a, Assessment of Operational Effectiveness 


With the proposed inclusion of a dedicated OT-3 period, the OT&E 
authority will need to develop tests for a free play, scenario based environment 
specifically designed to assess the COIs against the User Requirements. The 
analysis of these tests may be different from the weapons analysis currently 


performed. 


b. Assessment of Operational Suitability 


The assessment of operational suitability, especially for the RAN’s 
current combatant projects, would focus on interoperability and compatibility as 
these present the highest risks with the integration of proven equipment in a unique 
combination in a ship platform. RM&A data is collected and analysed during OT-3 
to give an indication of performance prior to AINS, and then continued for 12 
months during the OT-4 phase. The data would continue to be collected and 
analysed on an “as required”, rather than on a continuous basis by the Trials 
Authority until the RAN develops its routine in-service RM&A data collection and 
analysis system. The analysis of RM&A data is a specialist task, currently outside 
the experience of most uniformed personnel. To achieve significant and valid 


results, an improved RM&A analysis capability would be established. 


c. OT&E Training 


A formal course such as the USN Operational Test Directors Course 
should be a pre-requisite for key billets within the OT&E authority until the RAN 


develops enough knowledge and expertise to develop its own course. Other 
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authorities also require training in OT&E. The OT&E authority should develop 
acquaint courses for other personnel with a need to understand OT&E. Also to 
increase the awareness of OT&E in the RAN, the OT&E authority would deliver 
presentations regularly to Project Directors Courses, the ILS Management and 
Acquisition Course (ILSMAC), Defence management courses, and other 
appropriate acquisition courses. 

The proposed change of the head of the OT&E authority to a Warfare 
Officer as COT&E would provide a career path within the authority. As the 
credibility of OT&E increases within the warfare community, the knowledge and 
experience gained by warfare officers in this area will be seen to be of more benefit 
in future warfare related positions. The proposed employment of COT&E under 
COMEFLOT should also be more career enhancing for warfare personnel than the 
current arrangement under CSO(E). 

The professionalism and knowledge of T&E personnel could be 
enhanced by membership in a related professional organisation. The corporate 


membership of a suitable professional organisation may be worthwhile to the RAN. 


J. RECOMMENDATIONS 

It is recommended that the proposed OT&E model be reviewed, evaluated and 
implemented in the RAN to achieve an effective OT&E system to support the 
introduction of operationally effective and operationally suitable new combatants 


into the RAN. 


1. OT&E Definitions 


It is recommended that: 
a. the current definitions and phases of ship T&E be amended to better suit 


the Australian Defence environment. 


o/ 


b. the current TSTP be revised to show DT&E and OT&E events as 
separate categories of testing, and these trials be termed collectively as 


Integrated Tests and Trials (ITT). 


2. OT&E Documentation 


It is recommended that: 
a. the RAN T&E DI(N) and ABR 1921 be revised to achieve 


comprehensive and consistent T&E policy guidance. 


3. OT&E in the Acquisition Process 


It is recommended that: 

a. OT&E be conducted, where possible, prior to the production or 
purchase decision in the acquisition process to minimise risk. 

b. OT-3 be separated from the DT&E /PAT&E post delivery trials. 

c. OT-3 be conducted as a dedicated, free play, scenario based period after 
the completion of SQT and ORE to ensure testing is conducted on a 
worked up and materially proven ship. 


d. OT-3 of the overall ship be conducted as a complete weapons system. 


4. OT&E Organisation 


It is recommended that: 

a. the appointment of a DoD OTE policy maker and overseer within 
HQADF be investigated. 

b. DGNPW be appointed as the OT&E policy maker and overseer within 
the RAN. 

c. a close working relationship on T&E matters be required between 


DGFD(Sea), DGNPW and the OT&E authority. 
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ge 


DGNER be appointed as the Materiel Division T&E agency to assist 
Project Directors in complying with policies, incorporate lessons 
learned from previous projects, review the T&E aspects of projects at 
major milestones, and advise projects how to prepare for OT&E. 

the current CTE organisation be disestablished. 

an “In-service Trials” team be established under CSO(E) to conduct 
CTE’s current DT&E/PAT&E responsibilities, and those already the 
responsibility of CSO(E). 

Commander, OT&E (COT&E) be established as the RAN’s OT&E 
authority under COMFLOT, to manage OT&E and the post delivery 
trials period for all ships from the MHQ perspective. 


OT&E Policy 


It is recommended that: 


a. 


policy be developed to enable the performance in the Australian 
environment of foreign equipment to be evaluated efficiently, 
effectively and consistently. 

the TEMP approval process be reviewed to ensure appropriate 
authorities are consulted and approve the TEMP. 

the TEPG be chaired by DGNPW as the authority appointed to ensure 
the adequacy of the RAN’s overall T&E program. 

OT&E not be influenced by the Project or DAA, and that Project or 
DAA personnel not be present during testing. 

contractors not be present during OT&E. 

TEMPs for non-development projects be approved before production 
approval to enable inclusion of post delivery PAT&E/DT&E and OT-3 
in the Project Budget, and OT-4 in MHQ budget. 


Pages 


g. OT&E be conducted during a dedicated period in the LBTS following 
successful DT&E, to gain an estimate of the potential operational 


effectiveness and suitability of the system. 


6. Ship Acceptance 


It is recommended that: 
a. the Acceptance Board remain as a board of review. 


b. the Acceptance Board be convened for the first of class only. 


7. OTS&E Procedures 


It is recommended that: 

a. The OT&E authority develop tests for a free play, scenario based 
environment specifically designed to assess the COIs against the user 
requirement. 

b. analysis requirements of these tests be assessed, as they may be 
different from the weapons analysis currently performed. 

c. experienced personnel assist in designing suitability tests and analysing 
results. . 

d. interoperability and compatibility be a major focus during the 
assessment of operational suitability, especially for the RAN’s current 
combatant projects. 

e. an improved RM&A analysis capability be established. 

f. RMé&A data continue to be collected and analysed on an ‘as required’ 
basis until the RAN develops its routine in-service RM&A data 
collection and analysis system 


OT&E reports be made solely by the OT&E authority. 


ge 


OT&E phase reports be issued only, with no progress or interim reports, 


to allow full analysis and evaluation of the test results. 
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the USN Operational Test Director type course be a prerequisite for key 
billets within the OT&E authority until the RAN develops enough 
knowledge and expertise to develop its own course. 

the OT&E authority develop acquaint courses for other personnel with 
a need to understand OT&E. 

the OT&E authority deliver presentations regularly to Project Directors 
Courses, the ILS Management and Acquisition Course (ILSMAC), 
Defence management courses, and other appropriate acquisition 
courses, so to increase the awareness of OT&E in the RAN. 

a Warfare Officer be appointed to head the OT&E authority. 

. establishing corporate membership of a suitable professional T&E 


organisation be investigated. 


ou 





I. INTRODUCTION 


A. BACKGROUND 


1. Introduction 


Friday, May 15, 1992 marked the end of an era for Operational Test and 
Evaluation (OT&E) in the Royal Australian Navy (RAN). The RAN Trials and 
Assessing Unit (RANTAU), the RAN’s Operational Test and Evaluation authority, 
ceased to exist as an independent unit and was integrated with Maritime 
Headquarters (MHQ) under the Commander, Test and Evaluation.[Ref. 56]This 


marks a continuing shift in the emphasis placed on OT&E within the RAN. 


2. Field of Study 


The fundamental purpose of Test and Evaluation (T&E) in a system’s 
development and acquisition program is to identify the areas of technical risk to be 
reduced or eliminated. Operational Test and Evaluation is the means to demonstrate 
that a ship meets the specified user requirement before being Accepted into Naval 
Service (AINS). This study analyses USN and RAN Ship OT&E organisation, 
methodology, and procedures, and then compares and contrasts the two systems. 


From this analysis, a model for OT&E in the RAN is proposed. 


3. Importance of Research Effort 
In September 1991, the Chief of Naval Staff (CNS) expressed concern that 


the RAN may not be capable of adequately managing the comprehensive and 
complex procedures necessary to accept the ANZAC frigates and Collins class 
submarines into service [Ref. 62]. With these new classes of surface and sub-surface 


combatants under construction, and the RAN currently reorganising its OT&E 


organisation, the RAN could benefit from a timely analysis of USN and RAN Ship 
OT&E. The study is designed to assist the RAN to develop an effective and efficient 


OT&E organisation. 


B. OBJECTIVES 


OT&E is conducted to determine a system’s operational effectiveness and 
operational suitability, to identify system deficiencies and the need for potential 
modifications to meet established operational test thresholds, and to develop tactics. 
It is usually conducted at the end of each phase of the acquisition process to support 
a major milestone decision. The more significant of these is the OT&E conducted 
on a prototype to support a production or purchase decision, and on the first of class 
to support an Acceptance into Naval Service decision (AINS). OT&E is conducted 
against the current user requirement and is definitive feedback to the program’s 
sponsor that the system has met the user requirement. 

OT&E has a high level of representation within the U.S. Department of Defence 
and the USN. The Director, Operational Test and Evaluation, has direct 
responsibility to the Secretary of Defence for prescribing policies and procedures 
governing the conduct of OT&E in the Department of Defence, and is a member of 
the Defence Acquisition Board. The USN has a dedicated command, the 
Commander Operational Test and Evaluation Force (COMOPTEV FOR) 
responsible to the Chief of Naval Operations (CNO) for Ship OT&E. 

The RAN is currently reorganizing it’s OT&E authority. Prior to May 1992, the 
RAN Trials and Assessing Unit (RANTAU) was responsible to the Naval Warfare 
Branch, as the originator of user requirements, for the planning, conduct and 
reporting of OT&E. However, RANTAU has recently been disbanded and the 
OT&E component transferred to the RAN Maritime Commander - the eventual ship 


“user”. 
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Problems were experienced in the RAN with a perceived lack of high level 
representation and consideration of the OT&E requirements during the early phases 
of programs, with the cost and time effectiveness of OT&E, with the conduct of 
OT&E and Developmental Test and Evaluation (DT&E) concurrently, and with 
differences between acceptance of an item from a contractor (Delivery) and 
Acceptance into Naval Service. 

The RAN is a comparatively small navy with limited resources. With large 
indigenous programs including the new Collins class submarines and ANZAC class 
frigates due to undertake OT&E in the near future, it is imperative that the RAN has 
a cost effective and efficient OT&E organisation. 

This study compares the USN and RAN OT&E systems and highlights 
strengths and weaknesses of each. The methodology and procedures currently used 
by each Navy for the assessment of both operational effectiveness and operational 
suitability are analysed. The lessons learned from OT&E of the most recent USN 
ship program, the DDG-51, and the RAN DDG Modernization and Australian 
Frigate programs are reviewed. Then, using the results of this analysis, the current 
RAN plan for the OT&E of the ANZAC class frigate and Collins class submarine 
are reviewed. The output of the study is a model for the organisation and conduct of 


Ship OT&E in the RAN. 


C. THE RESEARCH QUESTION 


The primary research question posed in this thesis is “What are the fundamental 
characteristics of USN ship OT&E and how may USN experience in OT&E be 
applied to achieve an efficient, effective OT&E organisation for the RAN?” 
However, to address the fundamental characteristics of OT&E, a number of 
subsidiary research questions need to be answered. Some of the possible questions 


follow. How does OT&E support the Defence acquisition process? What are the 


fundamental characteristics of the USN OT&E organisation? What are the 
fundamental characteristics of USN OT&E policy? What are the fundamental 
characteristics of USN OT&E procedures? How is USN OT&E implemented? How 
does OT&E support the acceptance of ships into the USN Fleet? How do the USN 
and RAN OT&E systems currently compare? What needs to be done in the RAN to 
achieve an effective OT&E system with a minimum of additional resource 


requirements? 


D. SCOPE, LIMITATIONS AND ASSUMPTIONS 


The scope of this thesis is limited to the study of OT&E for naval warships and 
submarines. Although based on the same philosophy, the OT&E of aircraft and 
equipment is not addressed. Although other nations conduct OT&E (e.g., United 
Kingdom), this study is limited to the analysis of USN and RAN OTE systems 
only. This study is an overview of OT&E organisation, methodology and 
procedures. It does not address specific implementation matters or details of 
individual tests or trials. This study is limited to UNCLASSIFIED material only, 
and the collection of RAN information was limited to written and verbal 
correspondence as no personal interviews were able to be conducted with RAN 
authorities. No substantial increase of RAN OT&E resources in the near future was 


assumed. 


E. WHY STUDY USN OT&E? 
Study of the USN OT&E system could be beneficial to the RAN for a number 


of reasons. Due to the RAN’s long association with the USN and its purchase of U.S. 
ships and equipment, it has a similar philosophical approach to acquisition and 
T&E. The basis of much of the current RAN T&E policies, methods and procedures 
is derived from the USN. 


Technology is changing the face of modern warfare, and the USN is at the 
leading edge of this movement, adapting the best of new technologies to proven 
concepts and platforms. The USN strives to ensure that, as technology evolves, it is 
used to best advantage. As OT&E determines a new system’s operational 
effectiveness and operational suitability, the need for potential modifications to 
meet established OT thresholds, and develops tactics, it is a critical part of adapting 
technology to modern warfare. 

Being the most modern and capable naval force in the world today, the USN has 
substantial weapons system acquisition, operational and combat experience. This 
experience has confirmed their need for OT&E, and refined their OT&E 
organisation, policies, procedures and methodology. The RAN could benefit from 
this experience. 

The USN are world leaders in the development of high technology weapons 
systems and have extensive experience in their acquisition, testing and operation. As 
the RAN has a similar philosophical approach to acquisition and T&E, the USN 
OT&E system forms a good basis from which to develop an RAN OTK&E 


organisation, policies, procedures and methodology. 


F. RESEARCH METHODOLOGY AND LITERATURE REVIEW 


A balanced research methodology was used following the general research 


guidance outlined in Reference |. The research consisted of three strategies: 


1. Archival Research 
Archival research consisted of examining the relevant formal documents 
pertaining to OT&E in both navies. This included a review of U.S. Defence 
Acquisition law, followed by review of the U.S. DoD and USN official 


documentation, pertaining to T&E, and to ship OTE in particular. A similar 


approach was taken in reviewing the Australian documentation. Also, a literature 
search was conducted to elicit the latest published views and opinions on OT&E. 


This search was conducted through two primary sources: 


a. Database Search 


A search was conducted with DIALOG Information Services, a 
commercial information service available through the Dudley Knox Library, Naval 
Postgraduate School. The DIALOG service is a computer database accessible by 
modem. The search was conducted using a number of key words including 
Operational and Evaluation, policy. procedures, and destroyer and frigate. A listing 


of 200 recent items was provided [Ref. 2]. 


b. Defence Logistics Studies Information Exchange 


The Defence Logistics Studies Information Exchange (DLSIE), located 
at the U.S. Army Logistics Management College, Fort Lee, Virginia, isa U.S. DoD 
organisation that acquires, organises, stores and disseminates logistics and 
management information on a DoD wide basis [Ref. 3]. A custom bibliography was 
requested on the subject of ship Operational Test and Evaluation. A listing of 48 


studies and reports on the subject was provided [Ref. 4]. 


Other sources of information included journals of professional 
organisations and Defence magazines, however, the most useful guidance to 
relevant publications was provided by personnel involved in OT&E. In summary, 
the archival research established the current, formal organisation, methodology and 
conduct of OT&E in both navies, and identified the latest public opinions on OT&E 


issues. 


2. Personal Interview Research 

Ship OT&E is a highly specialised, evolving field of expertise. Many 
developments are not published formally by DoD or in the commercially available 
literature. It was necessary, therefore, not only to survey the literature on the subject, 
but also to conduct interviews. This research was conducted by personal interviews, 
either in person or over the phone, with key participants in the ship OT&E field. 
Views and opinions were sought from authorities in both navies who either conduct, 
use the results of, or are otherwise effected by ship OT&E. The list of pertinent 
references as well as personnel interviewed are contained in the References section 


of this thesis. 


3. Research Analysis 


Research analysis was then applied to the information gathered by the other 
two strategies, to assess the information, evaluate the strengths and weaknesses of 
each system, and to develop the OT&E model. This analysis consisted of breaking 
down the USN and RAN OT&E systems into their component parts, in order to 
determine how (and where possible, why) the systems are organised, and particular 
procedures and methods used. An OT&E model for the RAN was then developed 
from the results and conclusions of the research analysis. As the RAN has a similar 
philosophical approach to acquisition and T&E to the USN, their OT&E system 
forms a good basis from which to further develop RAN OT&E organisation, 


policies, procedures and methodology. 


G. DEFINITIONS AND ABBREVIATIONS 


Terms important to the analysis of OT&E systems in the RAN and USN are 


defined as they are introduced. A list of acronyms is included in Appendix B. 


H. ORGANISATION OF THESIS 


The study initially reviews the background of OT&E in Chapter II. The purpose 
of T&E in Defence acquisition is discussed and the types of T&E applicable to ships 
are introduced. The U.S. and Australian strategic, economic and Defence 
environments are discussed briefly and compared. Some similarities and differences 
between the USN and RAN are also noted. The philosophy, organisation and 
management of OT&E within the USN and RAN are then analysed in Chapters III 
and IV respectively. A comparative analysis of USN and RAN OT&E systems is 
included in Chapter V. As result of this analysis, a model for OT&E in the RAN is 
proposed in Chapter VI. Chapter VII details the conclusions and Chapter VIII lists 


the recommendations. 


Il. OPERATIONAL TEST & EVALUATION BACKGROUND 


This chapter presents background information considered necessary for a better 
understanding of OT&E. The purpose of T&E in Defence acquisition is discussed 
and the types of T&E applicable to ships are introduced. The U.S. and Australian 
strategic, economic and defence environments are discussed briefly and compared. 


Some similarities and differences between the USN and RAN are also noted. 


A. PURPOSE OF T&E IN DEFENCE ACQUISITION 

The fundamental purpose of T&E in a defence system’s development and 
acquisition program is to identify the areas of technical risk to be reduced or 
eliminated. During the early phases of development, T&E is conducted to 
demonstrate the feasibility of conceptual approaches, to minimise design risk, to 
identify design alternatives, to compare and analyse trade-offs, and to estimate 
operational effectiveness and suitability. As a system undergoes design and 
development, the emphasis in testing moves gradually from developmental test and 
evaluation, which is concerned chiefly with the attainment of engineering design 
goals, to operational test and evaluation, which focuses on questions of operational 
effectiveness, suitability and supportability. [Ref. 18:p. 1-1] A primary contribution 
made by T&E is the identification and reporting of deficiencies that may adversely 
impact the performance capability, availability or supportability of a system. 

T&E serves a number of useful functions, providing information for the 
following customers: 


e Developers to assist in the identification and resolution of technical 
difficulties. 


e Decision makers responsible for making the investment decision to procure 
a new system and for deciding on the most effective use of limited resources. 


e Operational users to support the development of effective tactics, doctrine 
and procedures. [Ref. 18:p. 1-1] 


B. TYPES OF SHIP TEST AND EVALUATION 


The principal types of T&E pertaining to ships are: 


1. Developmental Test and Evaluation (DT&E) 


DT&E is conducted to assist the engineering design and development 
process and to verify attainment of technical performance specifications and 


objectives [Ref. 11:p. 3]. 


2. Production Acceptance Test and Evaluation (PAT&E) 


PAT&E is testing conducted on production items to ensure systems meet 
contract specifications and requirements, usually for contractual acceptance 


purposes. It is a type of DI&E [Ref. 11:p. 5]. 


3. Operational Test and Evaluation (OT&E) 


OT&E is conducted to determine a system’s operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 
modifications to meet established operational thresholds, and develop tactics [Ref. 


11:p. SA]. OT&E has three distinguishing characteristics: 


¢ It is conducted in an operationally representative environment. 


e Itis conducted on production representative equipment using fleet personnel 
for operation and maintenance. 


e Itis conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 
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4. The Difference between DT&E and OT&E 


DT&E is focused on a system meeting technical and production 
specifications, while in OT&E, the focus is on assessing the actual functioning of the 
system in the realistic combat environment, against the user requirement. Figure | 


illustrates this principal difference between DT&E and OT&E. 


User Technical Manufacture 
Requirements Specifications of System 





Figure 1: DI&E and OT&E Comparison 


DT&E is planned and monitored by the developing agency and is normally 
initiated by the contractor. It includes the T&E of components, sub-systems and 
hardware and software integration, as well as preproduction and production 
qualification testing. OT&E, on the other hand, is conducted by an organisation that 
is independent of the developer, on the complete system in as operationally realistic 
an environment as possible, with hostile forces representative of the anticipated 
threat and with typical users operating and maintaining the system. Table | 


highlights the differences between DT &E and OT&E. 
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TABLE 1: DIFFERENCES BETWEEN DT&E AND OT&E [Ref. 18:p. 3-5] 


DT&E OT&E 
Controlled by Program Manager Controlled by Independent Agency 
One-on-One Tests Many-on-Many Tests 


Stenle Controlled Environment Tactical Environment with Operational 
Scenano 


Contractor Involvement No Contractor Involvement 
Trained Experienced Operators Users Recently Trained on System 


Specific Performance Measurements Operational Effectiveness and Operational 
and Goals Suitability Performance Measurements 


To further highlight the difference between DT&E and OTE, this 

















anecdote, originally from the Air Force Manual 55-43, 1979 and reprinted in the 
U.S. Defence Systems Management College Test and Evaluation Guide [Ref. 18:p. 


3-3] is an account of what is probably the first operational test and evaluation: 


The test and evaluation of aircraft started with the contract awarded to the 
Wright brothers in 1908. This contract specified a craft which would lift two 
men with a total weight of 350 pounds, carry enough fuel for a flight of 125 
miles, and fly 40 miles per hour in still air. The contract also required that 
testing be conducted to assure this capability. What we now call DT&E was 
satisfied when the Wright brothers (the developer) demonstrated that their 
airplane could meet those first contract specifications. However, no immediate 
military mission had been conceived for the Wright Flyer. It was shipped to 
Fort Sam Housten, Texas where Captain Benjamin D. Foulois, the pilot, had 
orders to “teach himself to fly”. He had to determine the airplane's 
performance, how to maintain it, and the kind of organisation that would use 
it. In the process, Captain Foulois subjected the Wright Flyer to test and 
evaluation in operational conditions. Foulois soon discovered operational 
deficiencies. For example, there was no seat on the airplane. During hard 
landings, Foulois’ 130 pound frame usually parted company from the airplane. 
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To correct the problem, Foulois bolted an iron tractor seat to the airplane. The 
seat helped but Foulois still toppled from his perch on occasion. As a further 
improvement, Foulois looped his Sam Browne belt through the seat and 
strapped himself in, Ever since then, contoured seats and safety belts - a 
product of this earliest “operational” test and evaluation - have been part of the 
military airplane. 


5. Operational Effectiveness and Operational Suitability 
A system’s operational effectiveness and operational suitability are the 
fundamental objects of OT&E. The USN and RAN definitions (which are possibly 


derived from the USN) for these terms are similar. 
a. Operational Effectiveness 


(1) USN Definition. Operational effectiveness is the capability of the 
system to perform its intended function effectively over the expected range of 
operational circumstances, in the expected environment, and in the face of the 


expected threat, including countermeasures where appropriate [Ref. 11:p. 8]. 


(2) RAN Definition. Operational Effectiveness is the capability of the 
system to perform its intended function to the required standard, over the 
expected range of operational circumstances, in the expected environment, and in 


the face of the expected threat including countermeasures[Ref. 44:p. A-1]. 


A subtle difference between these definitions is in the description of the 
assessed standard. The USN uses the term “effectively”, meaning to do the job 
regardless of meeting specification, whereas the RAN uses the term “to the required 


standard”, meaning to meet the specification. 
b. Operational Suitability 
(1) USN Definition. Operational suitability is the capability of the 


system, when operated and maintained by typical fleet personnel in the expected 
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numbers and of the expected experience level, to be reliable, maintainable, 
operationally available, logistically supportable when deployed, compatible, 


interoperable, and safe[Ref. 11:p. 9]. 


(2) RAN Definition. Operational suitability is the capability of the 
system, when operated and maintained by operational personnel in the expected 
numbers and of the expected experience level, to be reliable, maintainable, 
operationally available and logistically supportable [Ref. 44:p. A-2] in the specified 
environment within a specified time period.[Ref. 38:p. 14A-3] 


The relationship of operational effectiveness and operational suitability 


to OT&E, and the major questions addressed in OT&E are shown in Figure 2. 


OPERATIONAL 
EFFECTIVENESS 


DOES IT PERFORM 
AS INTENDED? 


OPERATIONAL 
TEST AND RELDIABIBIGY 
EVALUATION AVAILABILITY? 
MAINTAINABILITY ? 


OPERATIONAL SUPPORTABILITY ? 
SUITABILITY COMPATIBILITY ? 
TRANSPORTABILITY? 


IS IT SATISFACTORY INTEROPERABILITY ? 
FOR COMBAT USE? TRAINING? 
HUMAN FACTORS? 
SAFETY? 


DOCUMENTATION? 





Figure 2: Questions Addressed in OT&E [Ref. 18:p. 3-6] 
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C. AUSTRALIA’S DEFENCE ENVIRONMENT 


The recently publicly released “Australia’s Strategic Planning in the 1990’s” 
[Ref. 5] has not identified a specific military threat to Australia in the foreseeable 
future. It does, however, identify significant uncertainties concerning the future 


shape of the Australian strategic environment. 


1. Strategy 


Australia’s national strategic policy has evolved over recent decades. It has 
come from a position of defence dependence on allies through concentration on the 
immediate needs of self defence to a positive acceptance of both self-reliance and 
regional influence [Ref. 5:p. 3]. The latest Australian Defence Corporate Plan [Ref. 


8] highlights three central elements in Australia’s defence policy: 


e Self Reliance 
¢ Regional Cooperation 


e Strong Alliances 


Self reliance accords priorities to meeting credible levels of threat in 
Australia’s area of direct military interest [Ref. 6:p. vii]. Regional cooperation 
ensures strategic stability and security in the region, while strong alliances express 
common security interests and provide avenues for defence access and strategic 
influence. Close defence relations with the U.S. remain central to the policy of 
defence self-reliance. Although Australia does not depend on the U.S. for protection, 
it derives significant benefits from close collaboration through access to advanced 
technology, training and exercises, and intelligence. These benefits give Australia a 


level of capability that would be difficult and more costly for it to achieve alone. 


{Ref. 8:p. p 5] 
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2. Australian Defence Budget and Policies 


To achieve its defence goals, Australia’s defence budget for 1992-93 is $A 
9.8 billion. This accounts for about 9% of Federal outlays and is equivalent to 2.4% 
of the expected GDP. [Ref. 8:p. 23] Australia is placing emphasis on capital 
investment, devoting 27.3% of its defence budget. It intends to maintain capital 
investment at about $A 2.5 billion per year, of which about $A 2 billion is on capital 
equipment. Along with this significant investment in new capabilities, Australia is 
now buying far less “off the shelf” from overseas, with systems increasingly being 
designed and built in Australia, or adapted from overseas designs. [Ref. 8:p. 17] As 
a result, Australian is now building frigates and submarines due to enter service in 
the mid 1990's. 

The Australian Government’s forward estimates indicate that the Defence 
Budget will be reduced by 0.5% in 1993-94 and then maintained at this level in real 
terms in 1994-95 and 1995-96 [Ref. 8:p. 24]. Over the four years to 1996, reductions 
of up to 6,600 permanent military personnel (about 10% of total force) and 1785 
Defence civilians (about 8%) are anticipated [Ref. 8:p. 14]. 

The area over which the Australian Defence Force (ADF) needs to operate, 
the importance of denying an adversary freedom in the sea and air approaches, 
Australia’s limited population base and the characteristics of the northern Australian 
area, place a premium on technologically advanced systems rather than manpower 
intensive forces. As the costs of local development and production of 
technologically advanced systems are high, development emphasis is placed on 
those areas unique to Australia or where particular skills have ongoing relevance to 
a range of defence capabilities.[Ref. 5:pp. 28, 38] 

In summary, Australia is contributing to its strategic goals and ensuring that 


the ADF remains a formidable military force in regional terms, by maintaining 


16 


priority on capital investment. The focus is towards a smaller and more technology 
oriented force, with a more streamlined and effective organisation. Therefore, 
Australia needs to ensure that its more technology advanced systems are 


operationally effective and operationally suitable for the missions required of them. 


3. U.S. Versus Australia 


Of a similar physical size but much larger population, the U.S. is an 
economic and military superpower compared to Australia. Table 2 compares some 


significant characteristics of the two nations. 


TABLE 2: U.S. VS. AUSTRALIA 1991/92 [Ref. 10] 


| Population 2516 | 251.8million million 17.1 million 


($US 293.2 billion) 


Defence Budget $US 290.9 billion $A 9.44 billion 
($US 7.27 billion) 
% of Federal Outlay 22% 9% 


Total Armed Forces 1,913,750 


Spending on Capital | $US 71.7 billion $A 1.92 billion 
Equipment ($US 1.40 billion) 
% of Defence Budget 20.4% 


R & D Expenditure $A 0.22 billion 
($US 0.16 billion) 
2.3% 





































26.4% 
$US 36.2 billion 
















% of Defence Budget 12.4% 
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a. Differences 


The “National Military Strategy of the United States” [Ref. 7] affirms 
that U.S. Defence policy is built upon four key foundations: 


(1) Strategic Deterrence and Defence. The significant political 





instabilities in the former Soviet Union and the threat posed by the increasing 
numbers of potentially hostile states developing weapons of mass destruction 
requires that the maintenance of a modern, fully capable and reliable strategic 


deterrent remains the first defence priority. 


(2) Forward Presence. The day to day presence of U.S. forces in 
regions Vital to U.S. national interests is key to averting crises and preventing war. 
In addition to forces based overseas and afloat, forward presence includes periodic 
and rotational deployments, access and storage agreements, combined exercises, 


security and humanitarian assistance, port visits and military to military contacts. 


(3) Crisis Response. The capability to respond to regional crises on 
short notice is a key demand of U.S. strategy. This response may range from a single 
discriminate strike to the employment of overwhelming force to defeat a regional 


aggressor. 


(4) Force Reconstitution. Reconstitution is required to preserve a 
credible capability to forestall any potential adversary from competing militarily 
with the U.S. It involves forming, training and fielding new fighting units if required 
at short notice. It also involves maintaining technology, doctrine, training, 
experienced military personnel and innovation necessary to retain the competitive 


edge in decisive areas of potential military competition. 


This strategy is founded on the premise that the United States will 


provide the leadership needed to promote global peace and security [Ref. 7:p. 6]. It 
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does, however, reflect the shift from containing the spread of communism and 
deterring Soviet aggression to a more diverse, flexible strategy which is regionally 
oriented and capable of responding decisively to future challenges [Ref. 7:p. 1]. 
Although the U.S. and Australian strategic policy elements may not be directly 
comparable, the U.S. places a greater emphasis on Defence with strategies of world 
leadership, global influence and self sufficiency, while Australia places emphasis on 
regional influence and self-reliance. 

To achieve these goals, the U.S. devotes a considerably higher 
proportion of its Federal expenditure to Defence, than does Australia. It also spends 
a considerably greater percentage of its Defence Budget on R&D. U.S. policy is to 
maintain world leadership in key Defence technologies, by continuously improving 
the process of indentifying and introducing new technologies [Ref. 9]. Although 
R&D is conducted with emphasis on those areas unique to Australia, lower risk, 
proven technologies are preferred for the majority of new Australian Defence 
acquisitions. 

b. Similarities 

Like Australia, the U.S. current projections for FY 92-95 show a 
continued reduction of the Defence Budget by about 2% per annum. Additionally, 
with the recent change to a Democratic Administration under President Clinton, 
further reductions in Defence expenditure can be expected. During this period of 
budget and trade deficits and urgent domestic needs, the U.S. military strategy will 
be implemented within a significantly reduced Defence budget, and so places a 
premium on efficiency without compromising effectiveness [Ref. 7:p. 4]. 

As does Australia, the U.S. relies on technological superiority to offset 
quantitative disadvantages, to minimise risk to its forces, and to enhance deterrence 


and the potential for swift, decisive termination of conflict [Ref. 7:p. 10]. The U.S 
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also realises it can meet the challenges of the foreseeable future with a much smaller 
force than it has had in recent years [Ref. 7], and that implications of a 25% force 
cut by 1995 may be possible [Ref. 10]. Despite these funding and program cutbacks, 
the U.S. is also building new surface combatants and submarines. The first DDG-51 
“Arleigh Burke” completed testing and the first SSN-21 “Seawolf” is under 
construction. 

A significant similarity between the U.S. and Australia is one of 
culture. Both nations have predominantly caucasian, English speaking populations, 
a democratic system of government, close economic, military and political ties, and 
share similar cultural values and ideology. 

Despite being significantly larger than Australia in economic and 
military terms, and, consequently, having a different Defence strategy, the U.S. is 
similar to Australia in many areas. These include reducing budgets and size of the 
Defence force, a requirement for advanced technology, and a need for efficiency 
while maintaining effectiveness. But perhaps above all, a similar culture which 
promotes a relative ease of communication and assimilation of ideas, concepts, 


procedures and methodologies between the two nations. 


4. United States Navy Versus Royal Australian Navy 


A similar contrast applies to the navies of both nations. As Table 3 shows, 
the USN is far larger and more capable than the RAN. However, there are number 
of similarities. Although the RAN had its origins in the Royal Navy and with British 
systems, since the 1960’s the greater influence on the RAN has been from the USN. 
The purchase of three modified Charles F Adams (DDG-2) class destroyers from the 
U.S. in the mid 1960’s heralded an increasingly close relationship between the two 
navies. The purchase of four Oliver Hazard Perry (FFG-7) class frigates from the 


U.S. in the early 1980’s, and the local construction of two more of the class in 


Australia, continued this relationship. Although the RAN’s needs no longer support 
the purchase of current U.S. ship types, many new systems and equipment are still 
purchased through the USN. Ongoing support for these U.S. ships currently in RAN 
service, and Australia’s emphasis on interoperability with allies, ensure a continuing 
close association. The RAN shares with the USN much common equipment and 
weapons, compatible operational procedures, similar logistic support systems and 


defence acquisition procedures. 


TABLE 3: USN VS. RAN 1991 [Ref. 10] 
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Submarines 

Aircraft Carners 
Cruisers 

Destroyers 

Frigates 

Patrol (inc. Coast Guard) 
Mine Warfare 
Amphibious 

Support & Misc. 

Total Ships: 


Major Ship 
Acquisition Programs 
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DDG-51 class destroyers ANZAC class frigates 
SSN-21 class submarines COLLINS class subs. 


D. OT&E BACKGROUND SUMMARY 
OT&E forms a fundamental part of minimising risk in the weapons system 
acquisition process and in future system operations. It is conceptually different from 


DT&E, being focussed on assessing the operational effectiveness and operational 
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suitability of the system in a realistic combat environment, with respect to the user 
requirement. 

Australia is placing emphasis on capital investment for Defence, particularly for 
the RAN, with new frigates and submarines presently under construction. In 
response to budget reductions the current focus of the RAN is towards a smaller and 
more technology oriented force, with a more streamlined and effective organisation. 

The U.S. and Australia have a number of similarities, perhaps the most 
important, a similar culture which promotes a relative ease of communication and 
assimilation of ideas, concepts, procedures and methodologies between the two 
nations. The USN is a world leader in the development of high technology weapons 
systems and has extensive experience in their acquisition, testing and operation. 

With new frigates and destroyers due to enter service in the mid 1990s, the RAN 
has an urgent need for an efficient, effective OT&E system to ensure the operational 
risk of these new combatants is minimised, and the combatants meet the current 


requirements of the RAN. 
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Il. USN SHIP OPERATIONAL TEST AND EVALUATION 


This chapter describes the USN OT&E system. It explains the OT&E 
relationship and importance to the U.S. Department of Defence (DoD) acquisition 
process, the specific OT&E policies and methods employed in ship OT&E and why 
the OT&E of ships is different from other systems. The USN system of ship 
acceptance and the input of OT&E into the process is also described. The OT&E of 
the DDG-S1 class destroyer, the latest class of warship to undergo OTE, is used as 
an example of the implementation of USN OT&E policies. The workings of the 
USN system are well documented and the OT&E system can be traced from the 
statute requirements of Congress, through DoD directives to implementation by 


Navy specific instructions. 


A. U.S. DEPARTMENT OF DEFENCE ACQUISITION PROCESS 
The USN OT&E system forms an integral part of the US DoD acquisition 


process. In 1987, the Defence system acquisition process underwent revision in an 
attempt to make it less costly, less time consuming, and more responsive to the needs 
of the operational community. The Defence system life cycle consists of five phases 


of the acquisition process: 


¢ Concept Exploration/Definition 

¢ Demonstration/Validation 

e Engineering and Manufacturing Development 
e Full Rate Production/Deployment 

¢ Operational Support 


As Figure 3 shows, these phases are separated by major decision points 
(milestones) when a program is reviewed and authorised to advance to the next stage 
in the cycle, that provide a basis for progressive decisionmaking associated with 
program maturation. T&E results and planned T&E play an important part and are 
rigorously assessed as part of the milestone reviews [Ref. 18:p. 1-3]. These 


milestones are: 


¢ Milestone O - Concept Studies Approval 

¢ Milestone 1 - Concept Demonstration Approval 

¢ Milestone II - Development Approval 

¢ Milestone IIA - Low Rate Initial Production 

¢ Milestone III - Full Rate Production 

e Milestone IV - Major Modification Approval (as required) 
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Figure 3: U.S. Acquisition Process for Major Programs 


B. TYPES OF USN SHIP TEST AND EVALUATION 


The USN Instruction on Test and Evaluation [Ref. 11] defines the principal 


types of T&E pertaining to ships as follows: 


1. Developmental Test and Evaluation (DT&E) 


DT&E is conducted to assist the engineering design and development 
process and to verify attainment of technical performance specifications and 


objectives [Ref. 11:p. 3]. DI'&E is conducted in three major phases. 


2. Production Acceptance Test and Evaluation (PAT&E) 

PAT&E is testing conducted on production items to ensure systems meet 
contract specifications and requirements, and is a type of DIT &E [Ref. 11:p. 5]. The 
USN regard trials of new ships conducted by the Board of Inspection and Survey 
(INSURV) and shipyard industrial testing at PAT&E [Ref. 32:p. 2-5]. Two phases of 
ship PAT&E are defined: 


e Ship Construction Tests and Trials consist of all testing conducted on the ship 
during construction, including INSURV’s Acceptance Trials. 


e Ship Post-Delivery Tests and Trials are the conventional tests and trials, 
including INSURV’s Final Contract Trials, that commence after ship 
delivery and continue to the end of the SCN obligation or work limiting date. 


3. Operational Test and Evaluation (OT&E) 
OT&E is conducted to determine a system’s operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 


modifications to meet established OT thresholds, and develop tactics [Ref. 1 1:p. 


5A]. OT&E has three distinguishing characteristics: 
e It is conducted in an operationally representative environment. 


e It is conducted on production representative equipment using fleet personnel 
for operation and maintenance. 


US 


e Itis conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 


OT&E is subdivided into two major categories: 


¢ Initial OT&E (IOT&E) (OT-I and OT-II) which is up to and including 
OPEVAL (final phase of OT-2). 


¢ Follow-on OT&E (FOT&E) (OT-III and OT-IV) which is all OT&E after the 
final phase of OPEVAL. 


C. OT&E IN THE ACQUISITION PROCESS 


According to the Honorable Gerald A. Cann, Assistant Secretary of the Navy 


for Research, Development and Acquisition: 


Test and evaluation is the major control mechanism in the acquisition process 
for assessing system performance against system requirements. Navy decision 
makers rely on solid testing results, both developmental and operational, to 
provide the analytical groundwork for forming judicious program 
decisions. [Ref. 12] 


This statement by the USN’s senior acquisition executive, highlights the 
importance of T&E within the USN. Program advance from one phase to the next is 
not by the calendar of planned schedule, but by actual resolution of critical 
operational issues and achievement of pre-set thresholds verified by T&E [Ref. 
11:p. 2]. The importance of OT&E, in particular, to the progressive assessment of 
programs is such that its use has been mandated by the U.S. Congress and 


incorporated into the laws of the United States. 


1. Statute Requirement 


The requirement for adequate OT&E of Defence programs has strong 
congressional support and is mandated by law. The law includes the provision of a 


Director of Operational Test and Evaluation in the Department of Defence, 


appointed from civilian life by the President, by and with the advice and consent of 
the Senate [Ref. 16]. The law also addresses specific areas of OT&E reporting and 
conduct to ensure the Congress is kept informed, and the testing and reporting are 
impartial [Ref. 17]. By including the requirement that a major Defence acquisition 
program may not proceed beyond low-rate initial production (i.e. for Milestone III 
- Full Rate Production Decision) until Initial OT&E of the program is completed, 


Congress ensures that OT&E is an integral part of the acquisition process. 


2. OT&E Contributions at Major Milestones 


T&E progress is monitored by the Office of the Secretary of Defence 
(OSD) throughout the acquisition process. For major Defence acquisition programs, 
the Director, T&E and the Director, OT&E within OSD render independent 
assessments to the Defence Acquisition Board, the Defence Acquisition Executive 
and Secretary of Defence at each major milestone. These T&E officials also assess 
the T&E results for less than major defence acquisition programs that are 
specifically designated by OSD as OSD T&E Oversight Programs. The assessments 
are based on the following T&E information: [Ref. 18:p. 2-5] 


e The Test & Evaluation Master Plan (TEMP) and more detailed supporting 
documents developed by responsible Service activities. 


e Service operational test agency reports and briefings. 
e¢ Developmental test and evaluation data. 


a. OT&E prior to Milestone I 


During the Concept Exploration / Definition Phase prior to Milestone I, 
laboratory testing, modeling and simulations are conducted to demonstrate and 
assess the capabilities of key subsystems and components. The test and simulation 
designs are based on the requirements documented in the Mission Need Statement 


(MNS). The Operational Test and Evaluation Agency (OTA) monitors concept 
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exploration T&E to gather information for future test and evaluation planning and 
to provide effectiveness and suitability inputs desired by the Program Manager. The 
OTA also conducts operational assessments, as feasible, to assess the operational 
impact of candidate technical approaches and to assist in selecting preferred 


alternative systems concepts. [Ref. 18:p. 2-5] 


b. OT&E prior to Milestone IT 


During the Demonstration/Validation phase prior to Milestone II, 
concepts approved for demonstration and validation form the baseline that is used 
for detailed test planning. The OTA conducts early operational assessments to 
identify the best approach, indicate the risks and solutions for this phase of 
development, examine operational aspects of the systems development, and 
estimate potential operational effectiveness and suitability. This OT&E phase is an 
assessment only, with no actual testing being conducted. Typical operational and 
support personnel are used to obtain a valid estimate of the user’s capability to 
operate and maintain the system. The user of the system monitors test and evaluation 
during the concept Demonstration and Validation phase. Among the most important 
products of user monitoring are the attainment of early orientation and training, 
demonstrations of system performance and valid operational test assessments of 


systems maintainability and supportability.[Ref. 18:p. 2-7] 


c. OT&E prior to Milestone III 


The objective of the Engineering and Manufacturing Development 
(EMD) phase prior to Milestone III, is to design, fabricate and test a preproduction 
system that closely approximates the final product. T&E activities intensify during 
EMD, culminating in OPEVAL, and make significant contributions to the overall 
acquisition decision process. [Ref. 18:p. 2-9] OT&E conducted prior to the 


production decision is for the following reasons: 
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¢ Estimate the operational effectiveness and suitability of the system. 
¢ Identify operational deficiencies. 
¢ Recommend and evaluate changes in production configuration. 


¢ Provide information for developing and refining logistics support 
requirements for the system and training, tactics, techniques, and doctrine. 


¢ Provide information to refine operation and support cost estimates, and to 
identify system characteristics or deficiencies that can significantly impact 
these costs. 


¢ Determine whether the technical publications and support equipment are 
adequate. 


¢ Estimate the survivability of the system in the operational environment. 


d. OT&E after the Production Decision 
OT&E activities continue after the production decision in the form of 
Follow-on Operational Test and Evaluation (FOT&E) consisting of OT-III and OT- 
IV. It is accomplished to verify the operational effectiveness and suitability of the 
production system and to determine if deficiencies identified during initial OT&E 
(OT-III) have been corrected. A second phase of FOT&E (OT-IV) may be conduced 
by the user to refine doctrine, tactics, techniques and training programs over the life 


of the system. [Ref. 18:p. 2-10] 


D. US OT&E ORGANISATION 

Major participants in the US. OT&E process are listed in Table 4.The US OT&E 
community consists of a mix of service and OSD offices responsible for planning, 
programming, budgeting and evaluating operational tests. Figure 4 gives an outline 


of the OT&E organisation. 


TABLE 4: USN OT&E FUNCTIONAL AUTHORITIES 


OT&E Function USN Authority 
Policy Maker & Overseer Director, Operational Test & Evaluation (DoD) 










The US OT&E community consists of a mix of service and OSD offices 
responsible for planning, programming, budgeting and evaluating operational tests. 


Figure 4 gives an outline of the OT&E organisation. 
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Figure 4: U.S. OT&E Organisation 
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1. Policy Maker and Overseer 


The Director of OT& E (DOT&E) is the principal OT&E official in the 
Department of Defence and the principal advisor to the Secretary of Defence on 
OT&E. He is responsible for policy formulation, evaluation and oversight of OT&E 
within DoD. With status comparable to an Assistant Secretary of Defence, DOT&E 
is independent of other DoD officials and reports directly to the Secretary of 
Defence and additionally to Congress. His responsibilities include the following 


[Ref. 13:p. 7-4]: 


¢ Monitoring and reviewing all OT&E within DoD. 


e Designating observers to be present during preparation for and conduct of the 
testing portion of OT&E. 


¢ Controlling joint OT&E and coordinating OT&E conducted by more than 
one Military Department or Defence Agency. 


e Analysing the results of major system acquisition OT&E. For major systems 
and DOT&E oversight programs, reporting to SECDEF and to Congressional 
Armed Services and Appropriations Committees that OT&E is adequate and 
confirms effectiveness and suitability for combat systems tested. 


e Making recommendations to SECDEF on all budgetary and financial matters 
pertaining to OT&E, including facilities and equipment. 


¢ Approving OT&E plans for major Defence acquisition programs and 
DOT&E oversight programs. 


The office of DOT&E was established because, although each of the 
Services have had their own OT&E agencies, Congress perceived that operational 
testing was not as objective as it should be. It wanted a layer of oversight over the 


procurement process, especially OT&E. As a result, DOT&E also has reporting 


requirements to Congress as well as the Secretary of Defence. [Ref. 19] 
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2. Sponsor 

The Program Sponsor is responsible for acquisition program requirements 
and related system thresholds. The requirements are determined based on continuing 
assessments of current and projected capabilities in the context of changing military 
capabilities and national Defence policy. Within the USN, the Sponsor works for the 
Chief of Naval Operations (CNO). For major warship programs the sponsor is N8 - 
Warfare Requirements. Under the new USN organisation, increasing importance is 
given to the Fleet and Type Commanders in determining requirements and 
allocating priorities. Besides generating the user requirement, the Sponsor also has 


a review function in the OT&E process. [Ref. 20] 


3. Developing Agency 

When a program achieves Milestone 0, a program office is formed to 
manage the acquisition of the system. For ships and ship systems, this office may be 
established within the Naval Sea Systems Command (NAVSEA) or is affiliated with 
a Program Executive Officer, being ‘double hatted’, with responsibilities to both the 
Assistant Secretary of the Navy (Research, Development and Acquisition), and to 
NAVSEA. The program office is responsible for the conduct of DT&E in 
preparation for OT&E. To assist all NAVSEA programs with T&E matters, 
NAVSEA has established its own T&E office (SEA 60) whose purpose is to [Ref. 
235); 


¢ Assist Program Managers in complying with policies and incorporating 
lessons learned of the past. 


¢ Review T&E aspects of programs at major milestones 
* Keep COMNAVSEA and PEOs appraised of overall posture of T&E. 
e Influence OSD and Navy T&E policy. 


Although primarily a DT&E policy authority, this office also has an input 
into the OT&E process by reviewing the TEMP, and by conducting OT&E readiness 
reviews of NAVSEA programs. 


4. Operational Tester & Evaluator 


The Operational Test and Evaluation Force (OPTEVFOR) is the USN’s 
sole independent operational test agency responsible for the planning and conduct 
of OT&E. COMOPTEVFOR reports directly to the CNO, and is separate and 
distinct from the developing, procuring commands and the user. Results of 
OPTEVFOR evaluations are reported directly to CNO and to DOT&E. 
OPTEVFOR’s mission is to test and evaluate weapons systems, ships aircraft, and 
equipment in the anticipated operational environment and against the anticipated 
threat: to develop and validate procedures and tactics employing these weapons 
systems, ships, aircraft and equipment: and when directed by the CNO, to assist 
developing agencies in the accomplishment of DT&E [Ref. 14:p. 1]. OPTEVFOR 
involvement in early phases of research and development includes inputs to the Test 
and Evaluation Master Plan (TEMP), observing development testing, and 
conducting those phases or operational testing necessary to provide CNO with an 


early and independent operational assessment [Ref. 13:p. 7-5]. 


5. User 
The Commanding Officer and crew of the ship under test, and his Fleet and 
Type Commanders are the ultimate users of the system. The personal assessment of 
the Commanding Officer of the ship under test is sought by OPTEVFOR during the 
conduct of OT&E [Ref. 15:p. 44]. 
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6. Coordinator 


CNO has responsibility for ensuring the adequacy of the Navy’s overall test 
and evaluation program. T&E policy and guidance are exercised through the 
Director, Test, Evaluation and Technology Requirements (NO91). This organisation 
also acts as a T&E Focal Point and Coordinator, responsible for coordination of 
T&E matters in the designated Programs, System Commands and Department of the 


Navy. [Ref. 13:p. 7-5} 


E. USN OT&E POLICY 


USN OT&E policy has its basis in the statutes approved by Congress. This 
policy is refined in the DoD acquisition Directive 5000.1[Ref. 25] and DoD 
Instruction 5000.2 [Ref. 24]. The essential T&E elements in these directives are then 
further detailed in OPNAV Instruction 3960.10C [Ref. 11] and other subordinate 
instructions. This hierarchy of documentation establishes the disciplined 
management approach to OT&E taken by the USN. 


Basically, USN test and evaluation programs are structured to [Ref. 24:p. 8-2]. 


e Provide essential information for assessment of acquisition risk and for 
decision making. 


e Verify attainment of technical performance specifications and objectives. 
¢ Verify that systems are operationally effective and suitable for intended use. 
¢ Provide essential information in support of decision making. 


OT&E programs, in particular, are structured to determine the operational 
effectiveness and suitability of a system under realistic combat conditions and to 
determine if the minimum acceptable operational performance requirements as 
specified in the Operational Requirements Document have been satisfied. U.S. 


OT&E policy requires[Ref. 24:p. 8-5]: 
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¢ Threat representative forces be used whenever possible. 


¢ Typical users to operate and maintain the system under conditions simulating 
both combat stress and peacetime conditions. 


e Production or production representative articles be used for the dedicated 
phase of OT&E that supports the full rate production decision. 


The ultimate uses of information obtained through OT&E are feedback to the 
Developing Agency on the operational strengths and weaknesses of the system, and 
inputs for decisions to proceed with production and fleet introductions. A number of 


key factors of note are highlighted in USN OT&E policy: 


1. Independence 


The fundamental DoD Directive on Defence acquisition [Ref. 25:p. 1-8] 
requires an independent operational test activity. Each Military Department is 
required to establish an independent operational test and evaluation activity. This 
activity is required to: 

e Be separate and independent from the materiel-developing and procuring 


agency and the using agency. 


e Be responsible for planning and conducting operational tests, reporting 
results, and providing evaluations of each tested system’s operational 
effectiveness and suitability. 


e Report directly to the head of the DoD Component, except that the Secretary 
of a Military Department may delegate responsibility for supervising this 
activity to the Service Chief concerned. 


Acquisition managers are not to influence or attempt to influence the 
objectivity and completeness of test results presented to decisionmakers by the 
independent operational test activity. Independence is necessary so that there is no 
question of impropriety. The testing and reporting agency has nothing to gain or lose 


by whatever they say. As major acquisition decisions hinge on the results of OT&E, 


35 


independence is necessary to ensure that the reports on which those decisions are 


based are not self serving. [Ref. 19] 


2. Use of Contractors 


The use of system contractors in support of OT&E conducted to support a 
decision to proceed beyond low rate initial production (Milestone III) is restricted 
by law [Ref. 17] to ensure impartiality of testing. No person employed by the 
contractor for the system being tested may be involved in OT&E, unless contractors 


are planned to be involved when the system is deployed in combat. 


3. T&E Planning 


Test and evaluation planning is considered by the USN to be fundamental 
to the conduct of OT&E. The Test and Evaluation Master Plan (TEMP) is the 
controlling T&E document and is considered by some to be the single most 
important document associated with an acquisition program. The TEMP is directive 
in nature and defines and integrates test objectives, critical issues, system 
characteristics, test responsibilities resource requirements and test schedules. An 
approved TEMP constitutes direction to conduct the specified T&E.[Ref. 15:p. 6-1] 

Drafted by the Program Manager (PM), the TEMP’s main purpose is to 
combine the PM’s DT&E and COMOPTEVFOR’s OT&E into one integrated 
master plan. [Ref. 11:p. 11] OPTEVFOR draft Part Four on OT&E and provide 
OT&E resource requirements for Part Five. The TEMP is then co-submitted by both 
the Program Manager and COMOPTEVFOR. For ship programs, the TEMP is 
reviewed by the Assistant Chief of Naval Operations (Surface Warfare), the 
Director, T&E and Technology Requirements, and the Assistant Secretary of the 
Navy (Research, Development and Acquisition). The TEMP is ultimately approved 
at the DoD level by the Director, OT&E and the Under Secretary of Defence for 


Acquisition, Director Test and Evaluation. 
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Approval of the TEMP constitutes establishment of a contract between the 
requirement-setter and the PM which is essential to formal and accountable program 
execution, and between the requirement-setter and the OT agency, upon which 
COMOPTEVFOR will independently evaluate system operational effectiveness 
and suitability. An approved TEMP is required at Milestone I and is updated for each 
subsequent milestone and when significant program changes occur. The TEMP also 


serves several secondary purposes: 


e It allows all involved to see exactly what hurdles the system must clear. 


e It allows the PM to make good projections of COMOPTEVFOR’s OT&E 
costs, which must be funded. 


¢ It allows fleet, range, simulator and target schedulers to plan for the required 
services. 


4. Combined DT&E and OT&E 
The USN has a policy that combined DT&E and OT&E testing should only 


be considered when there are time and cost savings [Ref. 24:p. 8-3]. However, this 
combined approach must not compromise either the developmental or operational 
test objectives. Also a final independent phase of operational testing and evaluation, 
termed “OPEVAL”, is required for beyond low rate initial production decisions 
{Ref. 24:p. 8-4]. 

“Combined testing” refers to a single test program conducted to support 
both DT&E and OT&E objectives. The advantages of combined testing are the 
shorter time required for testing, and cost savings by eliminating redundant 
activities. These need to be weighed against the limitations of the additional 
extensive coordination required and the less than optimum environment and 
coverage for OT&E that may occur. Early involvement of OT&E personnel during 
system development increases their familiarity with the system and permits 


identification of operational concerns early in the program. 
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The U.S. has extensive experience with combined testing, some successful 
and some unsuccessful, and they conclude that it is possible to have combined test 
teams involved throughout the testing process. The DT&E and OT&E teams can 
share mutually beneficial data, as long as the test program is carefully planned, 
evaluated and reporting activities are conducted separately. [Ref. 18:pp. 17-1 - 17- 
4]. 


5. Modeling and Simulation 


DoD directives encourage the use of simulation and modeling to assist in 
projecting operational effectiveness and operational suitability prior to Milestone II, 
but limit their use in subsequent OT&E to that of supplementing OT&E test data. 
The use of modeling and simulation can increase the efficiency of the T&E process, 
reduce the time and cost, provide otherwise unattainable and unmeasureable data, 
and provide more timely and valid results. Although simulations are not a substitute 
for live testing, USN OT&E policy recognises that it is useful to augment tests by 
simulating non-testable events and scenarios, and for overcoming resource 
limitations. Simulation can also be used to extend test results, to improve the 
statistical sample, or to determine overlooked or directly unmeasured parameters. 


[Ref. 18:pp. 16-1 - 16-8] 


6. Foreign Weapons Evaluation 


The US DoD has a foreign weapons evaluation program which is designed 
to support the evaluation of a foreign nation’s weapons system, equipment or 
technology in terms of its potential to meet a specific U.S. military requirement. The 
primary objective of the program is to reduce the costs of U.S. research and 
development, while leading to the acquisition of foreign equipment for U.S. use 
[Ref. 18:p. 21-1]. From the OT&E viewpoint, the USN ‘try-before-buy policy’ is 


still maintained, despite the early phases of OT being unable to be achieved. 
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COMOPTEVFOR may be directed to assess the adequacy of any previously 
conducted OT&E and to provide recommendations on the need for additional U.S. 
T&E prior to procurement [Ref. 27:p. 6]. Obtaining pertinent test and evaluation 
data from foreign governments and manufacturers is necessary to preclude 


duplication and reduce costs. 


7. OT&E Funding 


Congress mandates that the costs of initial OT&E required to achieve a full 
production decision shall be paid from funds available for the system being tested 
[Ref. 17]. Funding associated with T&E (including instrumentation, targets and 
simulations) are identified in the system acquisition cost estimates, acquisition plans 
and the TEMP [Ref. 18:p. 18-11]. The Program Manager plans, programs, budgets 
and funds the costs of all resources identified in the approved TEMP for all T&E 
through OT-IIJ. OPTEVFOR estimates the costs to conduct OT&E and the program 
manager budgets and funds these costs. OT&E costs include test articles, 
expendables, targets, data collection and reduction and OPTEVFOR program 
related costs. The Program Manager does not fund fleet operating costs for T&E 
support, which includes fuel and aircraft, The operating costs for OT-II and III, and 
all costs for OT-IV, except procurement costs and OPTEVFOR costs, are funded by 
the Fleet CINCs.[{Ref. 11:p. 21] OPTEVFOR internal man-hours, and computer 


hours actually expended are not charged to individual programs. 


8. Land Based Test Sites 
A Land Based Test Site (LBTS) is a facility that duplicates, simulates or 
stimulates the employment of a system’s planned operational installation and 
utilization, primarily for the purpose of conducting DT&E. LBTS are often used to 
test system integration and overall performance. [Ref. 11:p. 12] COMOPTEVFOR 
advises the CNO on the adequacy of LBTS for the conduct of OT&E. LBTS test data 
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is normally used to support Milestone IIA decisions and not the more stringent 
Milestone III. Except where approved by CNO. OT&E intended to support 
production decisions will be performed in the operational environment in preference 


to the LBTS. 
9. Ship OT&E 


a. But Ships are Different 


The accomplishment of ship T&E varies considerably from the normal 
test cycle due to the lengthy period for design, engineering and construction of a 
major ship, and because ship T&E includes both that conducted on the ship platform 
itself, as well as that conducted on the equipment and systems to be installed on the 
ship. 

Ship acquisitions are low volume, high cost programs, and so, while 
subject to the same basic DoD and USN T&E policies applied to other systems, their 
procurement requires special T&E processes. If ships were procured the way other 
Navy systems are, the lead ship of the class would be used as a prototype for 
conducting T&E prior to approving the construction of the follow-on ships. Because 
of the time associated with the design and construction of a ship and the fact that few 
technical or operational risks are associated with the ship platform itself, the 
prototyping approach in not necessary. 

In a typical ship acquisition program, it can take about five years 
between the contract award for the lead ship and the time the ship 1s itself ready to 
conduct at-sea operational testing. To delay the construction of the follow-on ships 
would have a significant impact on the shipbuilding program’s costs and schedule. 
This longer schedule would substantially increase shipyard costs in return for a 
negligible reduction in the risk of the ship not meeting operational requirements. A 


significant time lag between the production of the lead and follow-on ships would 
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force a shipbuilder to let most of his experienced workers go and rehire and retrain 
others when construction resumed. Moreover, the significant cost savings available 
through quantity procurement of many of the ships’ equipments would not be 
realised if there was a time lag in production between the lead and follow-on 
ships. [Ref. 29] Also, from a contracting standpoint, a significant amount of lead 
time is required so that equipment will be available when needed for construction. 
Much of the long lead equipment is contracted before there is a ship available to 


test.[Ref. 19] 


b. Policies and Principles 


Congress accepted these practicalities of ship OT&E and amended the 
law to recognise the special case of ship OT&E [Ref. 30]. Because the development 
and construction period for a major ship normally precludes completion of initial 
OT&E on the lead ship prior to the production decision for follow-on ships, 
successive phases of IOT&E are accomplished as soon as practicable to reduce risk 


and minimise the need for modification to follow-on units. 


(1) The OT-I phase is an Early Operational Assessment (EOA) and is 
an evaluation of technologies, of processes, of plans and procedures, and design 
details to see if there any high risk areas that have a potential to cause cost overruns 


or degrade performance. 


(2) The OT-II phase for a shipbuilding program is an extension of this 
concept, except that it occurs in a time frame where landbased testing facilities or 
mock-ups of sections exist. Operability and suitability can be assessed, though not 


necessarily on a production representative item. 


Ship acquisition programs, therefore, usually have IOT&E between 


Milestones II and III, which consist of individual weapon systems testing and 
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system integration at land based test sites. A total ship OPEVAL is impractical in all 
cases except programs where a lead ship is constructed as an R&D effort involving 


a major technological advance in hull or machinery design [Ref. 28]. 


F. SHIP ACCEPTANCE 


1. Policy 


The purpose of ship acceptance in the USN is to ensure delivery to the Fleet 
of complete ships. free from both contractor and government responsible 
deficiencies [Ref. 35:p. 1]. Ship acceptance has traditionally been based only on the 
successful completion of PAT&E and material inspections, and for the first of class, 
has culminated in approval for Fleet Introduction. However, with the treatment of 
ships as complete systems, as for the DDG-51, Fleet Introduction was based also on 


OTE results. 


2. Definitions 


Reference 35 provides the major definitions for the acceptance process: 


¢ “Acceptance” is defined as the legal act of accepting custody of a new 
construction ship by the Navy upon delivery of the ship by a private builder. 


¢ “Delivery” is defined as the actual assumption of custody by the Navy 
incident to acceptance. The date of delivery from a private shipyard is also 
the date of acceptance. 


¢ “Fleet Introduction” is more a concept than a specifically defined milestone!. 
It signifies approval by the Secretary of the Navy that a ship class meets the 
operational effectiveness, operational suitability, safety and material 
standards for service use. 


1. Conversation between Captain Tobin, USN, Surface Division, Board of Inspection and Survey, 
and the author, 27 October, 1992. 


3. Responsibilities 


Independent verification of readiness of ships for acceptance is the 
responsibility of the President, Board of Inspection and Survey (PREINSURV) who 
conducts Acceptance Trials and Final Contract Trials for each individual new ship 
[Ref. 11:p. 10]. Based upon its findings, the Board recommends acceptance or final 
settlement of the contract. The Board identifies material conditions which represent 
departures from the General Specifications and deficiencies that substantially reduce 
the ship’s fitness for naval service and/or degrade its ability to perform its primary 
mission. [Ref. 34:p. 1] These deficiencies are noted in the Ship’s Log, but may not 
necessarily be corrected. 

A ship is normally accepted by the Commander, Naval Sea Systems 
Command. The preparation and presentation of a ship for acceptance trials is the 
responsibility of the Supervising Authority, normally the Supervisor of Shipbuilding 
(SUPSHIPS). Users also have a say in the acceptance process. A ship’s prospective 
Commanding Officer submits progress reports and can request changes be made 
which are essential to safety or the ship’s mission. The Type Commander monitors 
the construction and acceptance process to ensure “customer” input is provided, 
conducts a pre-commissioning habitability inspection and makes an acceptance 
recommendation to the Fleet Commander who, in turn, makes his recommendation 


on the request to deliver the ship.[Ref. 35:pp. 2 - 6] 


4. Procedure 


The procedure for ship acceptance and introduction to the Fleet is outlined 
in Figure 5. The Total Ship Test Program outlines two phases of this process [Ref. 
32:p. 2-6]: 


e Ship Construction Tests and Trials consisting of all testing conducted during 
construction, including Acceptance Trials. 


43 


¢ Ship Post-Delivery Test and Trials include final contract trials that 
commence after ship delivery and continue to the end of the ship construction 
obligation. 


ACTIVITIES 


Post - Final Post 
reba to a peice Shakedown |p Contract Shakedown 
rials Trials Availability 


MILESTONES | | 


A A A 


Acceptance Cc — Fleet 
/ Delivery eo Introduction 





Figure 5: Traditional Ship Acceptance Schedule and Milestones 


This procedure includes the following activities and milestones: 


a. Delivery / Acceptance 


Acceptance Trials are conducted by the INSURV Board when all work 
has been completed. These trials are conducted at sea and in port when the following 


prerequisites are met [Ref. 35:p. 11): 


e Successful completion of builders trials. 


¢ All installed equipment operable and capable of meeting performance 
specifications. 


¢ Habitability items complete. 
¢ Completion of surveys not requiring remote ranges. 
¢ Installation and checkout tests completed. 


¢ All applicable naval certifications completed. 


¢ Completed test reports and certifications available for inspection by the Trial 
Board. 


On successful trial completion, and on the recommendation of 
PREINSURV, COMNAVSEA accepts the ship on behalf of the USN from the 


contractor. 


b. Final Contract Trials 
Final Contract Trials are trials and materiel inspections conducted at 
sea and in port by the INSURV Board to determine if builder responsible 
equipments are operating satisfactorily [Ref. 35:p. E3]. It involves operation of all 
combat systems, and the propulsion system at full power. On completion, 
PREINSURV submits a technical assessment of readiness for OT&E to CNO, and 
to COMOPTEVFOR when tasked by CNO [Ref. 11:p. 10]. 


c. Fleet Introduction 

The successful completion of Final Contract Trials on the first ship of 
the class traditionally results in a recommendation for Fleet Introduction by 
PREINSURV to CNO. The Secretary of the Navy then approves Fleet Introduction 
of the ship class on the recommendation of CNO. In contrast, Fleet Introduction of 
systems is traditionally recommended by OPTEVFOR after successful completion 
of OT&E. However, for the DDG-51 class, where the ship was viewed as a complete 
system, Fleet Introduction was recommended by both PREINSURV and 
OPTEVFORafter the completion of OT&E. Figure 6 show the acceptance and 


schedule milestones used for DDG-51. 


2. Conversation with Commander J. Kren, USN, OPTEVFOR, and the author 29 October, 1992. 
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Figure 6: DDG-51 Ship Acceptance Schedule and Milestones 


5. Ship Acceptance and OT&E 
OT&E has traditionally not had an input into the ship acceptance process in 
the USN, with no involvement in Delivery, Acceptance or Fleet Introduction. It 
primarily focussed on the ship’s systems rather than the ship as a whole, and was 
fundamental to the Fleet Introduction of these systems. With the view of the DDG- 
51 as acomplete system, successful completion of OT&E is now a prerequisite for 


the Fleet Introduction of all new warship classes. 


G. USN OT&E PROCEDURES 

Procedures are the methodology by which policy is carried out. This section 
discusses the procedures adopted by the USN in implementing OT&E policy. 

1. From the User Requirement to the OT&E Test Plan 


The determination of the level and extent of OT&E required is a difficult 
question. For their multiwarfare capable ships, the USN focuses on the mission 


warfare areas to determine what is appropriate [Ref. 19]. The fundamental 
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cornerstone of OT&E is the user requirement, generated through a series of 
documents built into the acquisition process. The procedure involves progression 


through the following documents (Figure 7): 


e« Mission Need Statement (MNS) 

¢ Cost And Operational Effectiveness Analysis (COEA) 
¢ Operational Requirements Document (ORD) 

¢ Top Level Requirement (TLR) 

e Test and Evaluation Master Plan (TEMP) 

¢ OT&E Test Plan (OT&E TP) 


CONCEPT CONCEPT MAJOR 
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Figure 7: Progressive Definition of OT&E Documentation 
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a. Mission Need Statement 


Acquisition programs are based on identified mission needs that are 
generated as a direct result of continuing assessments of current and projected 
capabilities in the context of changing military threats and Defence policy. A broad 
statement of mission need, expressed in terms of an operational capability, not a 
system-specific solution, is identified in a Mission Need Statement (MNS). [Ref. 
24:p. 3-2] 


b. Cost and Operational Effectiveness Analysis 


A cost and operational effectiveness analysis (COEA) may be 
undertaken to evaluate the costs and benefits of alternative courses of action to 
satisfy the requirements of the MNS. The COEA looks at other alternatives e.g., are 
a lot of little ships or a few big ships needed to do the mission? The most suitable 
alternative is selected, e.g., a lot of little ships are required, and they need to do these 
things. It also develops the Measures of Effectiveness (MOE) for assessing the 


operational effectiveness of the alternatives. 


c. Operational Requirements Document 


From the MNS and COEA is developed the Operational Requirements 
Document (ORD) that contains the performance (operational effectiveness and 
operational suitability) and related operational parameters for the proposed concept 
or system. It contains the system performance objectives and minimum acceptable 
requirements to meet the requirements of the MNS. The MOEs used in the ORD and 
the technical parameters, are designed to support those same MOEs that were 


developed in the COEA [Ref. 19]. 


d. Top Level Requirement 
For some ship acquisition programs, Top Level Requirements are 
prepared after the MNS and ORD. This is necessary because of the length and 
complexity of the ship design process. The TLR states the high level requirements 
for the ship. For example, the TLR for the DDG-51, Section 2.11a Operational 
Directions, under Mission, Warfare Areas states: 
Simultaneous action in Strike Warfare, Antisubmarine Warfare, Antisurface 


Warfare and Antiair Warfare required. The combat system shall provide for 
rapid, accurate and efficient employment of the ship’s weapons. 


e. Test and Evaluation Master Plan 
The TEMP is developed from the ORD, and TLR if produced, and is 
the controlling T&E document that defines and integrates the test objectives, and 
Critical Operational Issues (COI) based on the ORD. For example, the DDG-51 
TEMP [Ref. 31:p. [V-2] highlights the COI of Combat System Performance as 
follows: 
Will the combat systems support simultaneous action against threats in Antiair 
Warfare (AAW), Antisubmarine Warfare (ASW), Antisurface Warfare 
(ASUW), and Strike Warfare (STW) missions during independent or 


combined operations as a unit of a Battlegroup, Surface Action Group, 
Underway Replenishment Group or Amphibious Task Force? 


This COI is addressed as a specific objective for OT-IIIA [Ref. 31:p. 
IV-11] which ts to: 
Determine the capability of the integrated DDG-51 combat system to detect, 


track, localise and engage threat representative targets in a multiwarfare 
environment (STW, ASW, ASU and AAW). 
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From this specific objective, a test procedure is outlined by 
OPTEVFOR. The level and extent of testing is based on the level of confidence to 
be assessed. It is usually a compromise between the extent of testing and funding 
available, agreed between the PM and OPTEVFOR with DOT&E oversight. The 
TEMP is viewed as a contract between the Developer, the Sponsor, OPTEVFOR 
and OSD, through DOT&E, as what is committed to be done and funding is 
available. Any change to the TEMP must be approved by all signatories. The TEMP 
identifies limitations to the scope of testing which may include threat replicators 
e.g., when a MIG-29 threat aircraft is needed during testing, but an F-14 will be 


used. [Ref. 19] 


ff. OT&E Test Plan 


This specific objective is then followed through to the OT&E Test Plan, 
developed by COMOPTEVFOR. The test plan contains “E” tests that assess 
operational effectiveness and “S” tests that assess operational suitability.The test 
plan then addresses how this will be done, by breaking it down into more specifics, 
AAW, EW, CAP, CIWS. Under DDG Test E-1 Combat System Performance, the 
Test Plan states: 

To determine the capability of the integrated DDG-51 combat system to 


detect, track, localize and engage threat representative targets in a 
multiwarfare environment (STW, ASW, ASU and AAW). 


g. Summary 
A requirement can, therefore, be traced from the mission need to the 
ORD, through to a COI and specific a objective in the TEMP, to the test plan and 
ultimately to the final OT&E report by OPTEVFOR. This linkage from mission 


need to operational test program defines the scope of testing. If a ship has a number 





of mission areas and there are MOEs to evaluate the performance in those mission 


areas, then a test program is developed around those MOEs. 


2. Assessment of Operational Effectiveness 


In assessing operational effectiveness, OPTEVFOR examines each COI 
and related operational effectiveness objective, and decides what needs to be known 
to enable each objective to be assessed [Ref. 15:p. 9-1]. For example, the objective 
may be “to determine the sonar’s capability to detect, classify, and track... in the 
natural acoustic environment.” OPTEVFOR may decide the following measures of 


effectiveness are needed to assess the overall objective: 


e Probability of detection 

e Detection range 

e Probability of correct classification given detection 
e Probability of classifying a threat as a nonthreat 

e Time between detection and classification 

¢ Classification Range 

e Probability of establishing a track, given detection 
e Time between detection and track establishment 

e Range at track establishment 


e Percent of time tracks are held 


They ensure that the appropriate environments, threats, etc., are included 
and that sufficient data will be generated to address the COI and objectives. 
OPTEVFOR initially determines the requirements and resources necessary to 
conduct the test. This is achieved by making a list of environments and other 
conditions under which the system will operate, and associating them with the 
various threats. This can be achieved by constructing a test design matrix of which 


Table 5 is an example. 
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TABLE 5: SAMPLE TEST DESIGN MATRIX [Ref. 15:p. H-2] 
eS GROUP TEST | BENIGN ECM CHAFF | SEASTATE | MULTI | TOTAL 
CHAR TARGET | ECM (SOJ/SSJ)_ | (SRBOC) (1-2) TARGET | MISSILE 
SEASKIM MOD A 3 
(<10m) BQM-34S B Cc Cc ABC 
SUBSONIC 
I 10-60m BQM-34S J E EJ 2 
IV SUBSONIC 
<30m MOD l 
SUPERSONIC | MQM-8G 
HH 30-60m MQM-8G H E FH 2 
VI SUPERSONIC 
I 60-300m MQM.-8G 
VI SUPERSONIC 


25-S0DEG MQM-8G 
DIVE 
SUPERSONIC 
10-60m 
SUBSONIC 


as SURFACE QST-35 



















The letters indicate missile firing runs in alphabetical order of priority. The 
columns are the environment in which the firing is scheduled. [Ref. 15:p. H-1] Then 
the actual resources available are reviewed to determine which test objectives must 
be eliminated due to lack of funding. 

Even though the operational test program is linked to mission need through 
the COEA, ORD and TEMP, there are a number of difficulties in developing 
appropriate OT&E tests. Many operational requirements are not very specific, 
meaning the testing and subsequent assessment of those requirements may be very 


subjective, and critical definitions need to be agreed before testing commences. 


a. Qualitative Vs. Quantitative Assessment 


The broad scope of many operational requirements makes their testing 
and subsequent assessment rather subjective, making the determination of 
meaningful and assessable quantitative MOEs difficult. Also a lack of resources 
may reduce an objective, quantitative, and statistically relevant test schedule to a 
more subjective, qualitative assessment. These situations require expertise and 
judgement, and compromise between the authorities involved, to make the tests as 
objective as possible. As an example, an antisurface warfare mission may require 
the ability to sink another ship, but the test program does not include this provision. 

The test program has to be designed so that it will provide confidence 
that the ability can be demonstrated without actually sinking a ship. Similar 
situations are encountered in all the warfare areas, and additionally with ship 
survivability. One objective is to achieve statistical relevance. DOT&E works 
closely with the USN to help determine how many test assets are required to achieve 
a certain degree of confidence that a test objective has been met. A test may need to 
be repeated a number of times to gain statistical relevance. However, if this is not 
possible, or the direct proving of an ability is not possible (as in the surface warfare 
example), then the test needs to be designed so that what is observed in the test 
program is projected to be what would be observed in reality. 


OPTEVFOR indicates the statistical significance of a result as follows: 


e “Determine” means a statistically significantly result with a specified level of 
confidence. 


e “Assess” means to qualitatively evaluate which will not achieve a 
statistically significantly result. 
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b. Definition Agreement 


Another difficulty in assessing operational effectiveness is in defining 
the meaning of the requirement and the measures of effectiveness. This is 
particularly applicable to whole ship testing which may incorporate a specific 
equipment requirement in addition to an overall ship requirement. For example, the 
definition of a “Detection Opportunity” for a sonar, may differ from what is an ASW 
‘Detection Opportunity” for the ship, given the other ASW related sensors that may 
be available. Agreement between authorities and the MNS is required for the whole 


ship versus individual systems. [Ref. 21] 


3. Assessment of Operational Suitability 


The assessment of operational suitability in the USN is more standardised 
than for the assessment of operational effectiveness. OPTEVFOR identifies 14 
suitability issues for which they have standard suitability tests [Ref. 15:p. 9-4 - 9-8]: 


¢ Reliability 

¢ Maintainability 

e Availability 

¢ Logistic supportability 
¢ Compatibility 

¢ Interoperability 

e Training 

e Human factors 

¢« Safety 

¢« Documentation 

e Transportability 

e¢ Wartime usage rates 
¢« Manning 


¢ Software supportability 


Software supportability has only recently been added by OPTEVFOR. It 
relates particularly to totally intensive computer systems, termed “software 
intensive systems”. It determines if a program manager has established adequate 
software support for his system.[Ref. 21] Not all these suitability issues need be 
addressed for a particular system. The nature and use of the system, and the phase 


of OT&E will determine the issues to be assessed. 


a. Suitability Assessment Procedure 


Early in the acquisition program, OPTEVFOR reviews the Integrated 
Logistic Support Plan (ILSP) against the ORD. The ILSP is a key acquisition 
document which defines the various methods used to provide the required range of 
logistic support for the system.Following ILSP review, OPTEVFOR determines the 
suitability tests required based on expected reliability, degree of confidence, 
thresholds, etc. required to be achieved. These determine the scope and length of the 
assessment. An onboard assessment at sea is conducted, during which 
questionnaires of crew members may be used and maintainability demonstrations 
are performed, usually using pre-faulted modules. Reliability, maintainability and 
availability (RM&A) data is collected and sent to the normal inservice analyst of 
RM&A data, the Naval Weapons Analysis Centre (NWAC) for analysis. In later 
OT&E, OPTEVFOR also uses data from the inservice maintenance management 
system (3M) and the individual system logs. [Ref. 21] 

Not only does OPTEVFOR have specialist analysts for effectiveness 
issues, it also has suitability analysts who design tests and determine measures of 
suitability. They also evaluate and analyse the adequacy of logistic supportability. 
OPTEVFOR also relies on operational personnel to use their experience and 


knowledge of the system to identify inadequate logistic support. 
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4. OT&E Reporting 


a. Reporting by COMOPTEVFOR 
COMOPTEVFOR’s evaluation report provides the CNO with 


conclusions regarding a system’s operational effectiveness and operational 
suitability, and his recommendations regarding the systems future, i.e., fleet 
introduction, further development, additional OT&E, etc. The report also contains 
the test results and evaluation criteria to substantiate the conclusions and 
recommendations. [Ref. 15:p. 12-1] A report covers a complete OT&E phase (e.g., 
OT-IIA), relating the test results to the COls and addressing the objectives stated in 
the TEMP. COMOPTEVFOR requests comments from the Commanding Officer of 
the ship under test ship. These comments are sent only to OPTEVFOR. All 
operational test data is considered to be the owned by OPTEVFOR until the final 
report is signed. Quick-look and interim reports are usually sent only if the testing 
could not be completed or when directed by the CNO. Although the final report may 
be a surprise to other authorities, the USN appears to favor this reporting procedure 


to ensure that: 


e the conduct of OT&E remains impartial and is not influenced by the program 
manager Or contractors during testing and analysis. 


¢ OPTEVFOR’s conclusions are based on a complete and thorough analysis. 
e OT&E results and analysis are reported through one authority only. 


b. Reporting by DOT&E 
The reports rendered by DOT&E address whether the OT&E 
performed was adequate and whether the OT&E results confirm that the system 
actually tested is operationally effective and operationally suitable. In the past it 
used OPTEVFOR reports as basis and provided their own judgement. Following 


General Accounting Office (GAO) criticism that DOT&E overall assessments 


56 


consistently presented a more favorable presentation to the Congress of test 
adequacy and system performance than was warranted by the facts [Ref. 36:p. 31], 
it now does an independent analysis from the raw test data collected by 
OPTEVFOR. As DOT&E does not have any specialist analysis staff, it hires an 
independent contractor who works exclusively for DOT&E, to do the analysis. 
Experienced operators at DOT&E then make the assessment. DOT&E’s written 
report to Congress is based now on OPTEVFOR reports and its own independent 
analysis.[Ref. 21] 


5. OT&E Coordination 

The USN recognises the importance of coordination between authorities in 
the successful achievement of T&E. The USN T&E coordinator is responsible to the 
Director, Test, Evaluation and Technology Requirements (NO91). He is essentially 
the OPNAV area coordinator for Navy T&E, providing a primary contact point for 
all parties and setting up T&E briefings and meetings. His responsibilities include 
the chairmanship of the Test and Evaluation Coordination Group (TECG) for each 
major program.[Ref. 37] TECGs are used for complex, multifaceted programs 
which require extensive T&E coordination. Membership of a TECG includes the 
Program Manager, the Sponsor, COMOPTEVFOR, a logistics coordinator and 
others as appropriate (such as a PREINSURV representative). TECG 
recommendations are considered for inclusion in the TEMP.[Ref. 1 1:p. 15] Some of 
the functions of a TECG are the early definition of terms, measures of effectiveness 
and how these are to be measured, and the criteria for acceptable or not acceptable. 
Of note is the formation of a TECG does not imply a joint test team approach. Each 
T&E agency remains fully and solely responsible for conducting and reporting the 


types and phases of T&E for which it is accountable. 
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6. OT&E Personnel Selection and Training 
OT&E is a specialised discipline, with its own philosophy and 
methodology. The U.S. community involved with ship OT&E select and train their 
military personnel. Although some personnel may develop into OT&E specialists, 
no career path in OT&E is consciously provided, and no naval personnel currently 


serving in DOT&E have served previously in OPTEVFOR. 


a. OPTEVFOR 


There are no special selection requirements for detailing military 
personnel for service as Test Directors in OPTEVFOR, although broad ship 
operations experience and combat system knowledge are preferred. OPTEVFOR is 
functionally organised along warfare lines, so specialists in each warfare area are 
preferred. Most Test Directors are of Commander rank, reporting to Captain level 
section heads. For particular programs, OPTEVFOR makes use of subject area 
experts, either resident or borrowed from a non-interested party. OPTEVFOR 
conducts an intensive four day Operational Test Directors (OTD) course covering 
the major areas of OT&E. Further details of this course are included in Appendix A. 
They also run adhoc segment courses which provide acquaints or updates on OTKXE 
subjects e.g., analysis, test plan development and threat updates. To assist their 
personnel in managing OT&E, OPTEVFOR publishes the OT&E Director’s Guide 
which documents its philosophy and methodology. [Ref. 21] 


b. DOT&E 
Military personnel assigned to DOT&E are selected by background and 


expertise, and are preferably war college graduates with joint experience. They are 
interviewed by their potential immediate superior, and by DOT&E himself, before 


their appointment is confirmed. DOT&E is platform, rather than functionally 
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organised, so specialist surface warfare, aviators and submariners are preferred. 
Although DOT&E does not conduct courses for new personnel, a one month 
handover is usually required. It does, however, publish a comprehensive staff 


orientation guide and some personnel have attended the OPTEVFOR OTD course. 
[Ref. 19] 


H. USN OT&E IMPLEMENTATION - THE DDG-51 PROGRAM 


The objective of the DDG-51 Program is to build the next class of destroyer to 
replace the aging ‘Coontz’ (DDG-37) and ‘Charles F. Adams’ (DDG-2) classes. The 
first of the class, the USS ARLEIGH BURKE has recently completed OT-III. It is 
the latest USN surface warship to undergo OT&E, thus serving as the most recent 
example of the implementation of USN OT&E policy and procedures. The DDG-51 
Program is under the management of the AEGIS Shipbuilding Program Manager 
(PMS 400) and is sponsored by the AEGIS Program Sponsor (N865G). PMS 400 is 
chartered to provide comprehensive direction and program management for all 
aspects of system development, ship acquisition and lifetime support preparation. In 
this task, PMS 400 has the traditional USN responsibilities of a ship acquisition 
program manager and also bears broadened responsibilities for the fleet introduction 


and lifetime support of the DDG-51 class ships.[Ref. 33:p. 1] 


1. Critical Issues 
The Critical Operational Issues for the DDG-51, as defined in the TEMP, 
are subdivided into effectiveness and suitability issues. [Ref. 31:p. [V-2] The 


effectiveness issues for DDG-S1 are: 


¢ Combat System Performance 
¢ Mobility System Performance 
¢ Command, Control and Communications Performance 
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e Support System Performance 
e Survivability 


¢ Tactics 


The suitability issues considered critical for DDG-5lare: 


e Reliability 
e Maintainability 
e Availability 


¢ Logistic Supportability 
¢ Compatibility 

¢ Interoperability 

¢ Training 

¢ Human Factors 

¢ Safety 

¢ Documentation 

¢ Software Supportability 


2. DDG-51 OT&E Program 


These COIs were assessed in a multi-phased OT&E program as shown in 


the schedule in Figure 8. 


a. Initial OT&E 


OT-II was split into two major system areas, Propulsion System and 


Combat System and was conducted at Land Based Test Sites. 


(1) OT-IIA Propulsion System Testing was conducted on the DDG-51 
Propulsion System at the Gas Turbine Ship Land-Based Engineering Site 
(GTSLBES), Naval Ship Systems Engineering Station (NAVSSES), Philadelphia, 
Pennsylvania. It is an Engineering Development Model that includes the Machinery 


Control System, Data Multiplex System and the engines, reduction gears, thrust 
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bearings and propulsion-related auxiliaries of the complete DDG-51 system. A 
water brake was used to simulate the propeller. The GTSLBS was exercised in a 
simulation of the operational environment of DDG-SIclass ships. Casualty control 
drills, maintenance demonstrations, and individual ship propulsion events were 
conducted during 424 hours of test operations. These included full power runs and 
emergency crash astern maneuvers. The system was assessed to be potentially 


operationally effective and potentially operationally suitable. [Ref. 31:p. [V-5] 
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Figure 8: DDG-51 Integrated Test Program Schedule [Ref. 31] 


(1) OT-IIB Combat System Testing was conducted at the Combat 
System Engineering Development Site (CSEDS) Moorestown, New Jersey. 


Following a philosophy of build a little, test a little, OT-IIB was conducted in three 
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phases. OT-IIB-1 was conducted in Jun 1986 on a partial configuration in an early 
stage of development and integration to support the program Milestone IIIA 
decision. The capability in all mission areas could not be evaluated with only search, 
detection and tracking of AAW targets conducted. No engagements of air targets 
were conducted. Twelve raids of multiple aircraft were conducted, of which eleven 
were in an ECM environment, including various jamming modes and chaff. 
Dynamic Test Targets (DTT) were also introduced to evaluate system performance. 
COMOPTEVFOR concluded that the combat system demonstrated the capability to 
control, integrate, and display information required for tactical decisions on AAW 
targets and recommended approval of the DDG-51 Combat System for limited fleet 
introduction. [Ref. 31:p. IV-4] 


(2) The second phase (OT-IIB-2) was conducted in September 1988 
when the CSEDS represented an operational prototype replicating planned human 
factors, sensors, information processing and weapons system installations in the real 
DDG-51. Assessment of capability in all mission areas was conducted. Thirty-four 
raids of manned aircraft were conducted, of which two only were in a clear 
environment. Additionally DTTs were also introduced to represent antiship missiles 
and surface hostile threats. Threat submarine evaluations, both active and passive 
were conducted as were Link 11 operations with aircraft, surface ships and shore 
stations. Non firing engagements with both Harpoon and Tomahawk were also 
conducted. Operational Suitability data were collected during the period. On test 
completion, COMOPTEVFOR recommended continued development of the 
Combat System and the conduct of an additional phase of OT-II at CSEDS prior to 


sea trials. 


(3) The third phase (OT-IIB-3) was conducted in August 1990 


following further development of the Combat System. The testing was 


comprehensive of the total Combat System’s capabilities. COMOPTEVFOR noted 
that performance was notably improved since the conduct of OT-IIB. They assessed 
the system as being potentially operationally effective and potentially operationally 
suitable, and recommended continued development of the Combat System. [Ref. 
31:p. IV-6] 

OT-IIB-2 and OT-IIB-3 were conducted to support continued limited 
production. DDG-51 crew members participated in the OT-II testing. both at the 
GTLBES at NAVSSES and in CSEDS testing. 


b. Follow On OT&E 


(1) OT-IIIA was conducted for a 28 day period in early 1992 on board 
USS ARLEIGH BURKE. Air, surface and subsurface targets were presented in 
single and multithreat environments. Simulated firings against manned aircraft and 
live firings against threat-representative targets in operationally realistic scenarios 
were conducted in order to evaluate the performance of the integrated combat 
system and the capability of the mobility and support systems to support the ship’s 
mission. The performance of all the ship systems designed for operation with other 
Navy units was also evaluated. Suitability tests, off-ship support and maintainability 


demonstrations were also conducted. [Ref. 31:p. [V-9] 

(2) OT-IIIB is planned to be conducted in a DDG-51 class ship to 
demonstrate correction of deficiencies identified during OT-IJIA and to include 
previously untested systems. 

(3) OT-IV will be conducted ina DDG-51 class ship to demonstrate 
correction of deficiencies, to complete deferred or incomplete OT&E, and to 
evaluate major computer software revisions as well as the DDG-51 class 


performance in battle group operations. 
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c. Timing of OT-II 
Table 6 shows the Post Delivery Schedule of DDG-51. Of interest is the 
timing of OT-III within the ship’s program. Note that OT-III is scheduled 
immediately after DT-III testing and before the Workup (REFTRA and FLEETEX). 
This has the disadvantage of OT&E not being conducted on a fully worked up ship 


with the crew still Developmental Testing oriented. This may have prejudiced the 


OT&E results. 


TABLE 6: USS ARLEIGH BURKE - POST DELIVERY SCHEDULE [Ref. 33:p. 5] 



















3. ‘Whole Ship’ OT&E 
OT-III conducted on the DDG-51 was unique in that it was the first time the 
USN has conducted OT&E on a whole ship in a multi-battle situation, in a fairly free 
play environment. Previous ship OT&E, e.g., on the “Aegis” class cruisers (CG-47), 
was essentially a combat system test based on a proven hull, where the rest of the 
ship and ship integration were not tested. DOT&E have been instrumental in 


intensifying efforts to create an operational environment. 


a. OT&E Program Development 


Following the guidance contained in the OTD manual [Ref. 15], 
OPTEVFOR developed a test design matrix. This matrix mapped the Top Level 
Requirements to the Schedule of Events (SOE). The TLRs were general and related 
to the warfare areas of AAW, ASW, ASUW and STW. The purpose of the matrix 
was to ensure that all TLRs were covered by an operational test. In the event that test 
resources became unavailable or a system casualty occurred, the matrix provided a 
reference of what had been accomplished, what remained to be accomplished, and 
the feasibility of completing the test objectives if OT was to continue. In the 
evaluation period, the “‘as-run SOE” was laid out chronologically, identifying events 
by mission areas. By examining the as-run table, individual mission areas were 
identified for analysis and periods of multiwarfare could be identified to look at 


synergistic effects.[Ref. 21] 


b. Integration of Engineering and Combat System Events 


Whole ship testing includes engineering and technical problems 
combined with combat system activity. The OPTEVFOR Operational Test Director 


controls the program and controls injection of incidents. A balance must be achieved 


65 


between engineering incidents and related degradation of the combat system to 


ensure the overall aim of the test is achieved. 


c. Operational Realism 


The OT&E emphasised operational realism by testing the ship as 
though in war, with a full political background to set tone and flavor. The situation 
consisted of an overall 28 day mission consisting of individual, but interlinked, 
scenarios. The scenarios were developed to examine the ship’s capability to operate 
as a fleet asset and to stress the installed combat systems’ capabilities against the 
threat. They were designed to provide a realistic threat in an operational 
environment and an element of surprise to provide realistic tests to support 
completion of the OT&E objectives and resolutions of the COIs. The Commanding 
Officer, Combat Information Centre team and the ship’s crew were allowed to 
respond to the tactical situation as they perceived it.[Ref. 31:p. IV-9] To enforce 
realism, operational message traffic flow at the same rate that could be expected in 


a conflict was maintained and Damage Control incidents were introduced. 


d. Limitations to Scope of Testing 


Despite the efforts for realism, certain practical considerations, time 
restrictions and resource constraints limited the scope of operational testing. These 


were [Ref. 31:p. [V-10]: 


¢ Targets and ECM did not fully replicate the threat in numbers and 
characteristics thus precluding fully threat representative raids. 


¢ The CIWS, ESM and ECM were not tested using drone targets. Manned 
aircraft were used for ESM and ECM tests and a towed target was used for 
CIWS firings. 


¢ The number of missiles available precluded reengagements and multiple 
salvo engagements in some scenarios. 


¢ The geographical location of test ranges and their facility and equipment 


limitations precluded fully threat representative target presentations, target 
density and open ocean geometries. 


¢ The duration of the operational testing precluded determination of reliability 
and other suitable parameters, forcing the use of development testing data 
where credible and applicable. 


¢ Operations were not conducted in all weather, temperature, humidity, and 
wind and sea conditions. 


e. DDG-S1 OT&E Analysis 
The OT&E of DDG-51 highlighted a number of overall 


considerations. [Ref. 21] 


(1) Interfaces. Interfaces are important to consider in whole ship 
testing, particularly individual systems with their own TEMPS and test plans. Whole 
ship OT&E looks at whole mission system and ship performance e.g., not just sonar 
performance but ASW system performance. Problems occur with concurrent sub- 
system and whole ship testing due to different critical issues and objectives of 
testing which create conflicts during testing. One overall test plan resolving these 


issues is required. 


(2) Statistical Significance. Some tests, e.g., Tomahawk firing, can 
only be done once. To achieve results of statistical significance, a number of 
simulation runs were conducted using new and varied scenarios. The one firing was 
then used to verify the simulation. This demonstrates the use of simulation to extend 


the test results. 


(3) Qualitative vs. Quantitative. Effort was made throughout DDG-51 
OT&E to use quantitative thresholds based on quantitative numbers, whereas, 
previous OT&E of the AEGIS class cruisers (CG-47) qualitatively assessed all 
COls. 
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(4) Qperationally Realistic Scenarios. To improve the measurement of 
AAW mission performance against a threat representative target in a threat 
operational scenario, particularly in the light of the “Iranian Airbus” issue, AAW 
scenarios now do not necessarily include everything that flies is hostile. To add to 
the operational flavor, a political situation and Rules of Engagement (ROE) are used 
during OT&E. These ROEs change due to the political situation or opposing force 
actions e.g., switching on fire control radar. This level of testing adds to the analysis 
problem e.g., having to cross the Electronic Warfare detection of a Fire Control 


radar to the aircraft, with the subsequent need for overall exercise analysis. 


(5) Crew Stress under Battle Conditions. The DDG-51 OT&E 
program include the measurement of crew stress under battle conditions, both 
during the land based and at sea testing. They observed that the land based level of 
operator performance was significantly better than at sea. The USN is continuing to 
pursue methods to take measurements to determine how much of the performance 
degradation from land based testing to at sea testing is a result of the personnel. The 
man machine interface, the concept of human factors and operational stress are a 
great contributor to overall system performance. One objective of the OT&E 


program was to keep the crew busy during the test period. 


(6) System Problems vs. People Problems. One aim of OT&E is to 


assess the human factors relating to system performance which involves assessing 
the adequacy of training, ease of use, etc. It does not include the performance 
assessment of an individual. This can sometimes be difficult when people form a 
vital part of the system. OPTEVFOR stress they are not drilling the crew, but 
demonstrate how the system (which may include people in the loop) responds. It is, 
therefore, important for the crew to be ‘worked up” to normal fleet standards prior 


to OT&E. 


(7) Independent Third Party Data. OT&E needs to be conducted ona 
range or with another ship/aircraft to get independent third party data to be able to 
reconstruct the evolution. The philosophy of testing is that the item under test is 
immediately suspect so if data from this system is required, then it needs to be 


verified. It may be sample verified so that 100% duplication is not required. 


(8) Program Manager Perception. Although many program managers 


may see OT&E as an annoyance and unnecessary burden to their program, PMS 400 
sees OT&E as an independent assessment to demonstrate that its on the right track 


and doing the right thing. [Ref. 22] 


I U.S. DEPARTMENT OF DEFENCE OT&E ISSUES 


U.S. DoD authorities involved in ship OT&E see a number of macroscopic 


issues relating to OT&E in the USN today. 
1. Impact of Budget Cutbacks 


a. Acquisition Strategy 

As a result of budget cutbacks, the Secretary of Defence is developing 
a new acquisition policy for systems that don’t go into a long protracted production 
phase. A prototyping acquisition strategy may be implemented where one of kind is 
built and the technology is either put on the shelf, or into limited production to keep 
defence industry active. More emphasis may be placed on R&D than on production. 
Questions arise as to how OTE fits into an acquisition strategy that does not lead 
to production, and should more reliance now be placed on DT&E and less on 
OT&E.[Ref. 19} Current thinking is that more importance may be placed on 


modeling and simulation to estimate operational effectiveness and operational 
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suitability of advanced technical demonstrators, to minimise the operational risk if 


production and deployment is required in an emergency. 


b. OT&E Conduct 


Questions arise also of how OT&E is to be conducted in a reduced 
budget environment. With reduced production runs, the relative cost of OT&E in 
relation to overall program cost will increase. This may not be seen as acceptable 
and pressure may be applied to reduce the absolute cost of OT&E. The breadth of 
assessment may be diminished using less test assets. There may be also a possibility 
of a central OT&E agency. With DOT&E becoming an increasingly aggressive 
participant in OT&E, testing in the future could be directed exclusively by 


DOT&E.[Ref. 19] 


c. Increasing Modeling and Simulation 
Modeling and simulation (M&S) capabilities are going to play a larger 
role in contributing to the OT process. Although M&S will never replace operational 
testing, it will be able to supplement actual testing to a greater degree in certain 
areas. Technological advances in M&S may make these tools now more of a 
candidate to be used for OT&E. Typically M&S is more of a study or analysis or 
developmental type tool, but are rapidly developing capabilities that will offer 


advantages to the OT process.[Ref. 19][Ref. 21] 


2. Operational Realism 
Following GAO cniticism of lack of realism [Ref. 36] and the ‘whole ship’ 
OT&E concept, test scenarios can be expected to be more realistic in the future. To 
obtain the maximum benefit from operational tests, they will also need to be more 
quantitative. This may include the development of integrated multiwarfare ranges to 


support whole ship testing.[Ref. 19][Ref. 21] 
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J. USN OT&E SUMMARY 


1. Importance of OT&E 

OT&E is the “final exam” in the U.S. DoD acquisition process, an 
empirical method of ensuring that sufficient technical return is made on acquisition 
investments and that a new system is fully capable of meeting the Fleet’s needs prior 
to production [Ref. 12]. As a result, the U.S. place high importance on OT&E. The 
requirement for the conduct of OT&E is mandated by Congress and incorporated in 
the laws of the U.S. OT&E is viewed as being more important as a basis for a 
decision to proceed beyond LRIP, than on the introduction into service of the final 
production item. This is because the “big bucks” of most Defence acquisitions are 
spent during the production phase. OT&E, however, continues to play an important 
role in the assessment of systems for Fleet Introduction and has increased its 


importance in the assessment of complete ships for Fleet Introduction. 


2. Need for OT&E 


The USN need for OT&E stems from the almost exclusive reliance by the 
U.S. on indigenous weapons development and production. They have a history of 
large, risky, developmental programs which push the state of the art in weapons 
technology. The outputs of these programs require the assessment of operational 
effectiveness and suitability before being committed to production and subsequent 
introduction into the fleet. The U.S. has been involved in a number of military 
conflicts which demonstrate the need for operationally effective and suitable 
systems. The lessons learned from Vietnam, and Gulf operations including the USS 
STARK and USS VINCENNES incidents, highlighted problems which DT&E 


alone may not identify. 
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3. OT&E Agency Independence 
Independence is the key to the effectiveness of OT&E in the U.S. 
acquisition process. OPTEVFOR is the sole independent OT&E agency within the 
USN, reporting directly to CNO, and DOT&E provides independent oversight and 
coordination of the military services’ planning and execution of OT&E, reporting 
directly to the Secretary of Defence. OT&E agency independence is designed to 
ensure impartiality, honest and open reporting, with a minimum of political 


interference. 


4. T&E Delineation 


The USN have a clear delineation between the different types of T&E. 
OPTEVFOR conduct only OT&E, and OT&E within the USN is conducted solely 
by OPTEVFOR. Thus, there appears to be good understanding in the USN of what 
is, and what isn’t, OT&E. 


5. Whole Ship OT&E 
Whole ship OT&E is conducted in a dedicated period following CSSQT 


and Final Contract Trials, and consists of fully assessed free play, multi-threat 
scenarios. The whole ship is viewed as an integrated warfare system, rather than as 


a platform for warfare systems. 


6. Clear Guidance 


The requirement for OT&E, and its organisation and methodology are well 
documented within the USN, providing clear guidance to acquisition and operations 
personnel. Also, training in OT&E is provided, further improving the capability and 


understanding of personnel involved with OT&E. 


7. Summary 


The USN OT&E system has, by necessity, developed into a comparatively 
well organised, well documented and effective, if complex, system that meets the 


requirements of the U.S. Defence acquisition process. 
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IV. RAN SHIP OPERATIONAL TEST & EVALUATION 


The recent disestablishment of the RAN’s independent Operational Test and 
Evaluation authority, the RAN Trials and Assessing Unit (RANTAU), and its 
subsequent integration with Maritime Headquarters (MHQ), marks a continuing 
shift in RAN OT&E policy. This chapter describes the RAN OT&E system, its 
relationship with the other categories of T&E and its role in the RAN ship 
acceptance process. As RAN T&E documentation does not yet reflect the 
disbanding of RANTAU, the OT&E system is described as it was prior to May 


1992, and then the new organisation will be introduced. 


A. AUSTRALIAN DEFENCE ACQUISITION PROCESS 


Acquisition of major capital equipment for the Australian Defence Force (ADF) 
is managed by the Capital Procurement Organisation (CPO) within the Department 


of Defence. The defence system life cycle consists of the following phases: 


* Conceptual 

¢ Demonstration and Evaluation 

¢ Full Scale Engineering Development 
« Production 


e In Service 


Similar, but not identical, to the U.S, system, these phases are generally 


separated by the following milestones: 


¢ Program Initiation 
¢ Full Scale Engineering Development 
¢ Production Approval 
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e Preliminary Acceptance (PA)/ Delivery 
e Acceptance into Naval Service (AINS) 
¢ Operational Acceptance (OA) 


FULLSCALE 
PROGRAM ENGINEERING PRODUCTION PRELIM — AINs 
INITIATION DEVELOPMENT APPROVAL ACCEPT 
APPROVAL 


DEMONSTRATION| FULLSCALE 
CONCEPTUAL AND ENGINEERING | PRODUCTION IN-SERVICE 
EVALUATION | DEVELOPMENT 


Figure 9: Australian Acquisition Process for Major Programs 





1. Development Vs. Procurement 


The completion of all acquisition phases applies only to indigenously 
developed systems. Systems procured overseas generally undergo only those phases 
from Milestone 3 onwards. Due to the RAN’s small size and limited requirements, 
it is not practical or achievable to develop all required systems locally. In many 
cases, systems with the fundamental capabilities required by the ADF are available 
overseas, however, some of these systems require additional development, 
adaptation or integration to meet Australian specific requirements. Many of these 
systems require only minimal adaptation (e.g., in their communications fit), while 
others are more extensive (e.g., ANZAC class frigates and COLLINS class 


submarines). This leads to different types of system acquisition programs. 
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2. Types of Acquisitions 


The following are the principal ways Australia acquires defence 


capabilities [Ref. 6:p. 86]: 
a. Indigenous Design and Development 


Local research and development can be undertaken to meet special 
local requirements, or as consequence of research undertaken to maintain the 


technology base. The new system is subject to all phases of the acquisition process. 


b. Local Development and Production 
Naval Adaptation programs in which an overseas design is purchased, 
but then undergoes modifications to meet specific RAN requirements. Some 


engineering development work may be needed. 


c. Local Production of Overseas Designs 


In some cases, the requirements of the ADF can be met most cheaply 
and expeditiously by existing overseas systems where there is no economic or 
technical prospect of a local competitor. The system may be built locally, where it 


would be subject to Production Approval. 


d. Import of Overseas Equipment 
Non development programs, where systems are purchased “off-the- 
shelf” from a foreign source, without modification, are subject to the acceptance 
milestones only. 
Each of these types of acquisition differ in their technical and operational 
risk and hence T&E requirements. In general, Australia has a policy of buying low 
risk, operationally proven systems, which may be modified to meet local 


requirements. 
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B. AUSTRALIAN DEFENCE T&E DEFINITIONS 

The categories of ship T&E used in the Australian DoD have been derived from 
the U.S. DoD instructions, but interpreted to accord with Australian requirements 
[Ref. 38:p. 14-1]. There is a hierarchy of documents pertaining to T&E within the 
DoD and RAN. 


1. DoD T&E Definitions 


At the DoD level, the Australian Capital Equipment Procurement Manual 
(CEPMAN 1) [Ref. 38] defines the types of T&E applicable to the Australian DoD 


as follows: 
a. Development Test and Evaluation (DT&E) 
DT&E assists the development specification, design and procurement 
process and to verify the attainment of development specifications. 
b. Production Acceptance Test and Evaluation (PAT&E) 


PAT&E is to assist the development of a procurement specification and 
to verify compliance with its requirements. It is generally associated with a 
production activity and normally forms the basis of qualified acceptance of the 


equipment. 


c. Operational Test and Evaluation (OT&E) 


OT&E is to assist the development of an operational capability 
specification and to measure compliance with its requirements. CEPMAN | also 
defines a sub-grouping within the OT&E classification - the Operational Evaluation 


(OPEVAL). An OPEVAL covers tests and evaluation on production representative 


Ti 


baseline equipment using the maintenance and support personnel and equipment for 


normal operational use and aims to: 


¢ Demonstrate operational effectiveness and suitability 
¢ Provide data to assist in the development of tactical aspects of the equipment. 


¢ Verify data, handbooks and documentation covering the operation of the 
system. 


2. RAN T&E Definitions 
These DoD definitions are further refined by the RAN. The Defence 
Instruction for RAN Test and Evaluation Policy [Ref. 44] defines the types of T&E 


as follows: 


a. Development Test and Evaluation (DT&E) 
DT&E denotes that T&E conducted in order to demonstrate the 
progressive achievement of design requirements and to show that the article will 


meet its technical specifications. DT&E is divided into three phases as shown in 


Table 7. 


TABLE 7: PHASES OF DT&E [Ref. 44:p. 2] 


TSE Te 
Validation of design concept 
DT-2 Proving design 


DT-3 Demonstration that production meets required technical 
characteristics 
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b. Production Acceptance Test and Evaluation (PAT&E) 
PAT&E denotes that T&E conducted by or on behalf of the 


procurement agency to determine whether contracted provisions have been 
satisfied. PAT&E is divided into two sections with a total of eight phases as shown 


in Table 8. 


TABLE 8: PHASES OF PAT&E [Ref. 44:p. 2] 


[a Ption and tains 
a 
ee 
earring 
2 












PAT4 Operational and Qualification Trials 
(conducted with OT-3) 

PAT-7 Follow on Operational T&E 
(conducted with OT-4) 


c. Operational Test and Evaluation (OT&E) 





OT&E is that T&E conducted to estimate the operational effectiveness 
and suitability by an authority independent of the development and production 
agencies. OT&E is divided into five phases as shown in Table 9. Of note is that 
Certification/Qualification trials are included in OT-3. These trials consist of 


Communications, Emitter and other individual system certifications. These trials 
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also include a five to seven week Combat System Ship Qualification Trial (CSSQT) 
that: 


¢ Demonstrates the maintainability and operability of those equipments/ 
systems that are included in the CSSQT program through accomplishment of 
equipment/system Planned Maintenance System (PMS), aircraft/balloon 
tracking exercises, practice missile firings and culminating in live firing 
exercises. 


¢ Provides training and familiarisation to ship personnel in the maintenance 
and operation of the installed equipments/systems that are included in the 
CSSQT program. 


¢ Identifies design problems in the equipment/system installed. 


¢ Identify any deficiencies that may exist in the CSSQT support elements, i.e., 
documentation, logistics, test equipment or training. 


TABLE 9: PHASES OF OT&E [Ref. 44:p. 2] 


r Tt~—=«SY's Operational assessment | Operational assessment of the development proposal tCtCt«éisd the development proposal 


Demonstration of achievement of program requirements for 
operational effectiveness and suitability of a prototype 
(OPEVAL) to support proceeding to full production 
























Demonstration of achievement of program requirements for 
operational effectiveness and suitability on production of ship/ 
aircraft/system, normally in independent operations using normal 
Fleet personnel. Certification/Qualification Trials include 
limited reliability, maintainability, availability and logistic 
Supportability assessments. 











Demonstration of achievement of program requirements for 
Operational effectiveness and operational suitability on 
production of ship/aircraft/system using normal Fleet personnel 
in a multi-force, multi-threat environment, Includes detailed 
reliability, maintainability, availability and logistic supportability 
assessments. 





Follow-on OT for assessment after modernisations, new 
applications or defect rectifications after OT-4 
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C. T&E IN THE ACQUISITION PROCESS 


1. The Need for T&E 


The most recent Australian Government “White Paper” on defence policy 


{Ref. 6:p. 70] states: 


We need to be able to determine the performance in our own environment of 
equipment of both overseas and local origin and to modify and adapt overseas 
equipment as necessary to improve its performance in our likely theatres of 
military operations. 

Also, the guidance for Australian defence acquisition [Ref. 38:p. 14-1] 


recognises that: 


The evaluation of military weapons systems and their individual component 
systems is essential to this policy. Analysis of the results of testing performed 
as part of the Test and Evaluation (T&E) of a capital equipment project may 
assist in the identification of problem areas, allow timely corrective action to 
be taken, and reduce the element of risk in major decisions. 

So the need for local T&E is recognised by the Australian Government and 
the DoD, for both indigenous and overseas systems. The responsibility for T&E of 
a system rests with organisations with responsibility for the design approval, 
certification or procurement of equipment, who have the authority and responsibility 
to conduct (or require the conduct of) T&E. All project requirements are coordinated 
by the Project Manager, who considers whether or not T&E can be conducted within 


available project resources.[Ref. 38:p. 14-2] However, the importance of OT&E in 


particular, is not specifically addressed in any of these T&E policy documents. 


2. OT&E Contributions at Major Milestones 


OTS&E in the RAN may be conducted during each phase in the acquisition 


process, but it is not essential to a local system meeting a development or production 
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milestone, or the local production or purchase decision of an overseas system. It is 


however, required before a system achieves Acceptance into Naval Service or 


Operational Acceptance. The phasing of T&E during the acquisition process is 


detailed in Figure 10. 


PROGRAM 
INITIATION 





FULLSCALE 
ENGINEERING 
DEVELOPMENT 
APPROVAL 


PRODUCTION 
APPROVAL 


PRELIM 


FULLSCALE 
ENGINEERING 
DEVELOPMENT 


Figure 10: Test & Evaluation in the Acquisition Process [Ref. 44:p. B-1] 


The contribution of OT&E to the acquisition milestones includes: 


a. OT&E prior to Program Initiation 


No role is documented for OT&E in the Conceptual Phase. 


b. OT&E prior to Engineering Development Approval 


During the Demonstration and Evaluation Phase, an operational 


assessment of the development proposal (OT-1) may be conducted. The results of 


which may contribute to the decision to undertake Full Scale Engineering 


Development. 


c. OT&E prior to Production Approval 


Towards the end of the Full Scale Engineering Development Phase, a 
demonstration of achievement of program requirements for operational 
effectiveness (OT-2) and suitability of a prototype (OPEVAL) to support 
proceeding to full production. This phase may include system/software assessment 
at a Land Based Test Site [Ref. 44:p. 2]. The results of OT-2, if conducted, may be 


used to support a local production or purchase decision. 


d. OT&E prior to Preliminary Acceptance 


No role is documented for OT&E in the Production Phase. 


e. OT&E prior to Acceptance into Naval Service 
Although Figure 10, indicates an OT-3 phase prior to AINS, no specific 
requirement is documented in RAN policy for the conduct of OT-3 prior to AINS. 
The prerequisites for AINS detailed in Reference 41 p 15.11, are described in terms 
of PAT&E Part 2, and make no mention of OT&E. At the working level, however, 
the RANTAU Standard Operating Procedure for OT&E [Ref. 53] includes the 
conduct of OT-3 prior to AINS, and the results of OT-3 have usually been 


considered in the AINS decision. 


f. OT&E prior to Operational Acceptance 


For first of class ships it will not normally be possible to trial and 
evaluate all aspects of the operational performance and maintainability prior to 


AINS. Therefore, the first of class usually undergo follow-on OT&E (FOT&E) after 


AINS, leading to Operational Acceptance [Ref. 44:p. A-1]. This OT-4 Phase 
consists of: 
¢ Evaluating the ship as part of a force during an exercise, in order to develop 


standard operating procedures and tactics, and to explore the limits of 
capability, etc. 


¢ Evaluating the maintainability and stores support for equipments and systems 
to determine whether or not the complement t, stores allowances, 
documentation are adequate. [Ref. 41:p. 15.14] 


g. OT&E Contributions post Operational Acceptance 


OT-S5 is follow-on OT&E for assessment after modernisations, new 
applications or defect rectifications after OT-4 [Ref. 44:p. 3]. 

The OT&E contributions to the Acquisition Milestones are by no means 
definitive. There is no documented requirement for the conduct of OT-1 and/or OT- 
2 prior to a production or purchase decision of major acquisitions. The decision to 
conduct OT&E being the responsibility of the Sponsor, Project Manager and the 
Defence Acquisition Committees. RAN T&E policy documentation almost 
exclusively focuses on the post production acceptance phases. Even during the 
acceptance phase, however, the requirement for OT-3 is clouded in definition 
differences between policy documents. The RAN appears to have its focus on 
Production Acceptance Testing, possibly because most systems procured in the past, 
were already in service with a large parent Navy. Tactical development was not 


necessary because the tactics also were acquired with the system. 


3. Production Acceptance Test and Evaluation 


As defined previously, Production Acceptance Test and Evaluation 
(PAT&E) is the testing conducted to demonstrate that systems meet contract 
specifications and requirements: also that items/systems are properly installed and 


operable onboard the ship. The objective of PAT&E is to confirm that the contractor 
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has met contractual obligations, and hence the Commonwealth may correctly take 
delivery of the ship. [Ref. 41:p. 14-2] Much emphasis is given within the RAN to 
PAT&E, known under various terms including Inspections, Tests and Trials (ITT), 
Harbour Acceptance Trials (HATs) and Sea Acceptance Trials(SATs). In recent years 
the RAN has adopted the U.S. Total Ship Test Program (TSTP) concept, initially 
with the modernisation of the DDGs, and now in the testing of the locally produced 
FFG-7 class, under the Australian Frigate Project. The adoption of the TSTP by the 
RAN included the redefinition of the test phases. Where the U.S. system included 
only PAT&E, the RAN has included elements of DI&E and OTE, in addition to 
PAT&E, and gave them PAT designations. 


a. Integrated Test Package 


Under the TSTP, the program and content of the formal ITT required 
by the contract is usually designated the Integrated Test Package (ITP). Although in 
the Australian system, Government testing is also included. The categories of ITT 


testing are listed in Table 10. 


TABLE 10: CATEGORIES OF ITT [Ref. 41:p. 14.3] 
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b. Overlap of PAT&E and OT&E 


PAT&E Part 2 covers the period from Preliminary Acceptance to the 
AINS and equates to PAT-6. The Project Director has responsibility for the program 
and its funding and RANTAU is responsible for the detailed schedule of events and 


the provision of test and trials resources. Testing includes: 


e Ship’s staff familiarization, shakedown and safety drills 

e Installation and testing of specialised equipment 

¢ Tempest test 

¢ Harbour trials of weapons, communication and combat data systems, etc. 
¢ Ship Qualification Trials 


e Degaussing and Noise Ranging 


However, some of these testing activities are also included as part of 
OT-3, and both PAT-6 and OT-3 are conducted by the same authority (RANTAU), 
during the same period, and using the same T&E personnel. Similarly, the PAT-7 
phase equates to OT-4. 


D. RAN OT&E ORGANISATION 


The RAN community responsible for planning, programming, budgeting and 
evaluating OT&E forms an integral part of the more general T&E organisation. 
OTQ&E in the RAN is not treated as being special or unique, and no organisation 
deals solely with OT&E. The RAN organisational structure has changed over the 


past few years which has impacted on the structure of the OT&E community. 


1. OT&E Organisation Elements 


The functional elements of an OT&E organisational structure include: 


¢ Policy Maker and Overseer 


¢ Sponsor 
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¢ Developing Agency 
¢« Tester and Evaluator 
e User 


¢ Coordinator 


a. Policy Maker and Overseer. 


There is no clear OT&E policy maker within the Australian DoD or the 
RAN. The Chief of Capital Procurement within DoD has a role in the formulation 
of capital equipment procurement policy for the guidance of Service Materiel 
Divisions. The CPO manual [Ref. 38] addresses T&E, but more in the form of 
guidance rather than policy. The Director General of Naval Warfare (DGNW) in the 
Deputy Chief of Naval Staff (DCNS) Division, drafted the RAN Test and Evaluation 
Policy Defence Instruction [Ref. 44]. DGNW however, is now defunct with the 
result that Test and Evaluation policy for Navy is presently in a state of flux, with 


no office having assumed the responsibility [Ref. 50]. 


b. Sponsor 

The sponsor is responsible for monitoring operational requirements and 
warfare concepts, the development of concepts for operations for naval warfare 
systems, and the raising of staff targets and requirements as required. He/She is also 
responsible for the subsequent trials and continuing overview to ensure that the 
equipment meets the requirement. Originally the responsibility of DGNW, this 
function for major acquisitions was assumed by the Director General Force 
Development (Sea) in Headquarters, Australian Defence Force (HQADF). This 
centralised force development and user requirements organisation produces the 
Naval Capability Proposals (NCP). The decision to subject an acquisition to T&E is 
initially decided by the sponsor and detailed in the capability proposal [Ref. 44:p. 1] 


c. Developing Agency 
When a project is approved, a project office is formed within the 
Materiel Division responsible for acquisition of the capital equipment to meet the 
approved NCP. The nature and extent of the T&E to be conducted will be decided 
by the Project Director in consultation with the Design Approval Authority and 
RANTAU and detailed in the TEMP [Ref. 44:p. 1]. The Project Manager, in 
consultation with operational, technical and maintenance authorities, is to fully 


investigate the necessity for, and likely scope of, Defence T&E [Ref. 38:p. 14-5]. 


d. Operational Tester and Evaluator 


The RAN Trials and Assessing Unit (RANTAU) was the RAN’s 
OT&E authority with a mission to provide an independent evaluation, audit and 
training service in response to current and future Maritime Force requirements. 


Located in North Sydney, near the main Fleet Base, the trials related functions of 


RANTAU included[Ref. 43:p. 1]: 


¢ Conduct of operational effectiveness and operational suitability trials on 
ships and facilities under construction, modernisation, conversion or 
extended refit in support of AINS. 


¢ Conduct test and evaluations, investigations and inspections of selected 
equipment when tasked. 


e Audit and witness selected Harbour Acceptance Trials (HATS), Sea 
Acceptance Trials (SATS) and Test Procedures which have operational 
implications for ships and facilities (now known as PAT 4 and 5). 


¢ Conduct Ship Qualification Trials. 


RANTAU, although the OT&E authority, also conducted PAT&E on 
behalf of Project Directors and/or the Maritime Commander. With the 


disestablishment of RANTAU in May 1992, the trials functions were transferred to 


the Commander, Test and Evaluation (CTE) under the Chief Staff Officer 


(Engineering) in the Maritime Command. 


e. User 
The Maritime Commander, and the individual ships’ Commanding 
Officers and crews are the ultimate users of a ship, RAN T&E documentation, 
however, does not address their involvement in T&E. The Commanding Officer of 
a ship under test usually provides RANTAU with his opinions on the ship's 


performance at the conclusion of a trials period. 


f Coordinator 
The Director of Naval User Requirements (DNUR), under DGNW, 
once acted as the Navy Office representative for RANTAU within Navy Office, 
Canberra. He provided a coordination/liaison role between RANTAU and the 
Materiel Division, by attending project meetings and highlighting T&E 


requirements. 


2. Changes to the RAN OT&E Organisation 


The OT&E organisation within the RAN has changed dramatically over the 
past few years. Figure 11 illustrates the OT&E organisation in 1987. The Director, 
RANTAU (of Captain rank) was directly responsible to the Assistant Chief of Naval 
Staff - Development (ACDEV-N). Also under ACDEV-N, the Director General 
Naval Warfare (DGNW) sponsored all major acquisitions, and gave T&E policy 
direction. Under DGNW, the Director of Naval User Requirements (DNUR), 
provided RANTAU representation and performed an OT&E coordination role 
within Navy Office. Thus the OT&E authority was responsible to the developer of 
the user requirement, and was independent of the developing and production 


agencies. 
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Figure 11: RAN OT&E Organisation circa. 1987 





By 1990, the structure outlined in Figure 12 had developed. With the 
centralisation of military policy development and resource planning. the 
Development Division within the RAN was disbanded. DGNW, with reduced user 
requirement responsibilities, moved under the Deputy Chief of Naval Staff (DCNS), 
The emerging HQADF now became the Sponsors for major acquisitions. The 
Director General Force Development (Sea) (DGFD(SEA)) having particular 
responsibility for “Sea” Capability Proposals [Ref. 54]. DGNW, however, retained 
responsibility for T&E policy within the RAN. RANTAU was also transferred under 


DCNS. As the sponsorship of major acquisitions was now held by HQADF, the 
OT&E authority was now no longer responsible to the developer of the user 
requirement. DNUR was dissolved and the OT&E authority lost its representation 


in Navy Office. 
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Figure 12: RAN OT&E Organisation circa. 1990 





SECDEF 







By the end of May 1992 (Figure 13), RANTAU was disestablished and its 
trials functions were transferred to the Maritime Commander as the Commander 
Test and Evaluation (CTE) under the Chief Staff Officer (Engineering) (CSO(E)). 


DGNW was dissolved and his now reduced functions were assumed by the Director 


2) 


Naval Warfare (DNW) under the newly created Director General Naval Policy and 
Warfare (DGNPW)[Ref. 55] within the DCNS Division. 
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Figure 13: RAN OT&E Organisation post May 1992 


So, as a result of reorganisations over the past five years, the RAN has no 
TRE, let alone OT&E, policy maker and overseer. It also has no OT&E coordinator 
within Navy Office and the RAN’s OT&E authority is buried within the in-service 
engineering management area of the Maritime Command. The OT&E authority, 


however, remains independent from the developing and production agencies. 


E. RAN OT&E POLICY 
RAN OT&E policy is embedded in the more general T&E policy. 


1. Policy Documentation 


The following documents outline the RAN T&E policy: 


a. Capital Equipment Procurement Manual (CEPMAN 1) 
Australian DoD T&E policy has its basis in the Capital Equipment 
Procurement Manual (CEPMAN 1)[Ref. 38], the aim of which is to assist project 
managers in assessing the need for T&E in a project and to provide guidelines for 
its conduct. The document provides broad T&E policy and, other than defining 


OT&E, little policy is provided as to its conduct. 


b. DI(N) LOG 82-1, RAN Test and Evaluation Policy 
The top level T&E policy document for the RAN is the Defence 
Instruction (Navy) on RAN Test and Evaluation Policy [Ref. 44]. Its purpose is to 
issue policy for the conduct of T&E in capital procurement and modernisation 


projects in the RAN. 


c. ABR 1921 
ABR 1921 [Ref. 41], contains T&E and acceptance policy applicable to 


ships building, undergoing modernisation, conversion or extended refit. Although it 


refers to OT&E under PAT&E Part 2, it does not address OT&E specifically. 


2. T&E Decisions 


RAN T&E policy [Ref. 44:p. 2] states that the decision to subject an 
acquisition to T&E is initially decided by the Sponsor and detailed in the Capability 
Proposal. The nature and extent of the T&E is then decided by the Project Director 
in consultation with the Design Approval Authority and RANTAU. Any conflict 
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over the requirement, nature and/or extent of the proposed T&E is resolved by 
DCNS and ACMAT-N for operational and design matters respectively. However, in 
practice, the nature and extent of T&E is broadly decided by the Sponsor, and the 
Project Director is responsible after consultation with the DAA and the Trials 


Authority [Ref. 45]. 


3. Independence. 


The independence of a testing authority is recognised in general terms in 
CEPMAN I [Ref. 38] which recognises that independent verification and validation 
(IV&V) is an important means of providing early detection of problems, and that it 
can have value and credibility. However, it implies that “independence” means 
independent from the Defence Department, i.e., contractors. With respect to OT&E, 
CEPMAN I compromises the independence of an OT&E authority by stating that 
the Design Approval Authority (DAA), which is part of the Developing Agency, is 
responsible for the assessment of the acceptability of the product design as a basis 
for Acceptance into Service. It provides for the DAA being involved in offering 
support to trials establishments (as required) during OT&E in order to provide DAA 
advice to the Project Manager on the suitability of the product for Acceptance into 
Service.[Ref. 38:p. 14-7] This appears to be more applicable to services other than 
the RAN, as the DAA within Navy does not advise the PM on the suitability for 
AINS. 

The RAN T&E Policy DI(N) [Ref. 44:p. 3] is more clear when it states 
OT&E is to be conducted progressively by an authority independent of the 
development and production agencies (usually RANTAU). With the demise of 
RANTAU and the transfer of OT&E responsibilities to CTE on the staff of the 
Maritime Commander, independence from the development and production 


agencies is maintained. 
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4. Use of Contractors 
Although the Australian definitions of OT-3 and OT-4 (Table 9) includes the 
requirement that the system should be demonstrated “using normal Fleet 
Personnel”, the documentation makes no policy statement regarding contractor 
personnel involvement in system operation or maintenance, during OT&E. It has 
been RAN practice (e.g., in the DDG Modernisation Project) to have contractors 
onboard resolving technical problems during the SQT period, which is included as 


part of OT-3 in the RAN. 


5. T&E Planning 
The RAN uses the Test and Evaluation Master Plan (TEMP) as the single, 


executive, long range planning document for T&E in a project. The TEMP is to be 
raised in draft form as early as practicable. Based on the USN format, it is the 
ultimate responsibility of the project sponsor because it is the primary tool to ensure 
the user requirement is met. However, during the period when there is a full time 
Project Of fice, TEMP coordination is normally carried out by the T&E Manager in 
the Project Office. RANTAU (now CTE) as the OT&E authority provides the OT&E 
input. The TEMP is endorsed by all affected organisations and is released jointly by 
the Deputy Chief of Naval Staff (DCNS), and the Assistant Chief of Naval Staff- 
Materiel (ACMAT-N), as the Developing Agency. If any issues remain unresolved 
at this level, the TEMP will be resolved by the Chief of Naval Staff (CNS). [Ref. 
44:pp. 4-5] The authority for DCNS to release the TEMP was based on his role as 
Project Sponsor. With HQADF now assuming this role, the DCNS function in 


TEMP development is unclear. 
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6. Combined DT&E and OT&E 


Although not specifically addressed as such, combined DT&E, (PAT&E) 
and OT&E is implied as being condoned and almost mandatory within the RAN. 
The T&E DI(N) [Ref. 44:p. 3] states that OT-3 and OT-4 will incorporate the 
requirements of PAT-6 and PAT-7 Inspection Tests and Trials as described in ABR 
1921. It also states that OT-3 will be conducted during Ship Qualification Trials. No 


limitations or guidance as to the possible hazards of this approach are addressed. 


7. Modeling & Simulation 

The use of models and simulation (M&S) in OT&E are not addressed by 
the RAN documentation, and their application to OT&E has been limited. 
Simulation and stimulation were used extensively during DT&E by both the 
Submarine Warfare Systems Centre (SWSC) and the Combat Data System Centre 
(CDSC) for the Submarine Weapons Update Program (SWUP) and DDG 
Modernisation respectively. However, Major combatant OT&E applications of 
M&S within the RAN have been limited to the DDG Modernisation Project. M&S 
developed by the Maritime Systems Division of the Defence Science and 
Technology Organisation were used by RANTAU for the evaluation of the upgraded 


combat and weapons systems in the Modernised DDGs. 


8. Foreign Weapons Evaluation 


Despite the most recent Australian Government “White Paper” on Defence 
Policy [Ref. 6:p. 70] requiring the need to be able to determine the performance in 
the Australian environment of equipment of both overseas and local origin, no DoD 
or RAN T&E policy specifically addresses the evaluation of foreign systems. 


Although, almost all of the systems procured by the RAN in recent years are 


“foreign” in origin, the RAN documentation focuses on the T&E process for the 


progressive evaluation of locally developed systems. 


9. OT&E Funding 


OT&E is funded separately within a project and is estimated and bid for by 
RANTAU to Project Directors. [Ref. 44:p. 6] 


10. Land Based Test Sites 


The guidance for OT &E using land based test sites includes a statement that 
system centres and simulators will be employed for early stages of OT&E if 
available [Ref. 44:p. 4], and that OT-2 may be conducted at a land-based test site 
[Ref. 44:p. 3]. 


11. Ship OT&E 


The RAN recognises that some major acquisition projects will be treated 
differently to others in so far as protracted construction time usually precludes T&E 
of a prototype before the decision to proceed to production. The degree of technical 
risk will be assessed in each case and, where considered sufficient, the decision 
taken to use land-based test sites and/or prove a particular system in another 
platform. When this occurs, DT-2 and OT-2 shall be conducted at the test site and 


the “whole ship” evaluation undertaken as soon as practicable after delivery. 


F. RAN SHIP ACCEPTANCE 


1. Policy 


The decision to accept a new or modified ship into the RAN is vested in the 
Chief of Naval Staff (CNS). An Acceptance Board is established to provide an 


assessment of the ship and submit recommendations upon which CNS can base his 
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acceptance decision. The assessment of a ship leading to an acceptance decision has 
two distinct components [Ref. 41:p. 15.7]: 
¢ Determining where the ship as built and tested does not meet the Agreed Ship 


Characteristics, contracted performance specifications and approved support 
standards. 


¢ Deciding whether those Agreed Ship Characteristics, performance 
specifications and support standards accord with current policies standards 
and practices. 


The Acceptance Board (AB) is an independent body established by CNS to 
make recommendations for CNS’s acceptance into RAN service of new 
construction or modernised ships, submarines, aircraft. and, where directed, new 
systems, installations and equipment. Usually established for each new ship type 
and only for first of class, a Board is composed of a President, a Vice President (VP), 
and additional members selected from appropriate specialist areas. The President, 
VP and Board members serve on the Board part time. The Board is administered by 
the Director, RANTAU (now CTE) who is an ex of ficio member, assisted by a full 
time Secretary.[Ref. 42:pp. 1-3] The AB’s sole function is to advise CNS on 
acceptance matters, and it is essentially a board of review. 


The RAN divides acceptance into three phases: 


¢ Preliminary Acceptance 
¢ Acceptance into Naval Service 


¢ Operational Acceptance 


2. Definitions 


The “Delivery Commissioning and Acceptance” chapter of Reference 41 


provides the definitions for the acceptance process: 


¢ “Delivery” occurs when the ship is contractually delivered from the 


98 


contractor’s ownership to the Government. 


e ‘Preliminary Acceptance” is an assessment by the RAN to determine if the 
ship has reached both material and support status sufficient to safely and 
effectively proceed with the Navy trials and evaluation program. It usually is 
concurrent with Delivery. 


e ‘Acceptance into Naval Service” (AINS) signifies the acceptance of the ship 
as a fully operational unit of the Fleet, based on operational performance and 
support levels obtained at that time. 


¢ “Operational Acceptance” is carried out only on the first of class to 
supplement the AINS assessment using additional operational and support 
experience gained with the Fleet to refine the vessel and Naval 
Requirements. 


3. Responsibilities 


Delivery of the ship to the Government is a contractual matter and is 
managed by the Project Director. Once delivered, the process is one of ‘offer and 
accept’ whereby the ship is offered by the Project Director to CNS for acceptance. 
RANTAU’s responsibility is to conduct the Acceptance Trials and report the results. 
The Acceptance Board does not conduct tests, trials or inspections, but may witness 


them if required. [Ref. 42] The Acceptance Board roles are as follows: 


¢ Be satisfied that a comprehensive and properly supervised series of 
inspections, tests, trials and evaluations have been carried out to the point 
where assessments can be made on current suitability for operational service 
on the basis of the approved current requirements. 


e Assess the ship, aircraft or equipment, by examining reports of materiel 
inspections, operational trials and evaluations, attending trials as required 
and calling for such other tests or evaluations as necessary. 


¢ Be satisfied that current, relevant and approved documents when 
promulgated describe the required operational characteristics against which 
the item for acceptance can be tested and evaluated for operational 
suitability, operational effectiveness and supportability in service. 


4. Procedure 


The completion process for a warship entering service is outlined in Figure 


14. This process may be divided into two distinct parts: 
* Those events leading to and culminating in delivery / preliminary acceptance. 


¢ Subsequent activities leading to AINS and Operational Acceptance (if first of 
Class) 


ACTIVITIES 


PAT-4/5 Dee PAT-6 / OT-3 
Trials Availability ine. SQT 


MILESTONES 


A A A 


Delivery / 
Preliminary 
Acceptance 


Acceptance into Operational 
Naval Service Acceptance 





Figure 14: RAN Ship Acceptance Schedule and Milestones 


a. Delivery! Preliminary Acceptance 


Delivery of the ship to the Government is a contractual matter and is the 
prerogative of the Project Director acting on behalf of ACMAT-N and the 
Commonwealth of Australia [Ref. 41:p. 15.3]. Immediately prior to delivery, the AB 
will advise CNS whether the ship has reached a materiel and support status suitable 


for Preliminary Acceptance. Preliminary Acceptance assures CNS that the vessel is 


considered safe for a Naval crew to take control of the vessel from the contractor and 
conduct outstanding RAN trials of equipment at sea or undertake specific trials for 


AINS assessment. [Ref. 41:p. 15.10] 


b. Acceptance into Naval Service 


Acceptance into Naval Service is essentially the acceptance/transfer of 
the ship from the acquisition part of the Navy (ACMAT-N) to the inservice 
Operations and support areas, as having met the user requirement. The AINS 
assessment is carried out on completion of the Post Delivery Availability (PDA) 
Trials Period, at which point the ship should be in a materiel state ready for 
operational service. Post delivery trials leading to AINS include the SQT period. 
with harbour and sea elements. While the PD is responsible for the PAT&E 
program, RANTAU is responsible for the detailed planning and witnessing of the 
trials program consulting with the PD. During these trials, the PD may provide a full 
time trials coordinator onboard to assist the Commanding Officer and the other trials 
authorities.[Ref. 41:pp. 15.10 - 15.12] On completion of the trials period, the AB 
will advise CNS whether the materiel and support state of the ship is sufficient for 
it to enter service as an operational unit of the Fleet, and hence it can be accepted 


into naval service [Ref. 42:p. 4]. 


c. Operational Acceptance 


For a first of class ship it will normally not be possible to trial and 
evaluate all aspects of the operational performance and maintainability prior to 
AINS without the duration of the PAT&E Part 2 program becoming excessive and 
because certain aspects can only be evaluated in the operational environment. This 
follow-on T&E period is conducted following AINS and is compiled by RANTAU 
in consultation with the Naval and Maritime Headquarters Staff. It comprises a 


series of trials and evaluations to assess operational effectiveness and operational 
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suitability. [Ref. 41:p. 15.14] Typical examples of these activities in the FOT&E 
program are: 
e the evaluation of the ship as part of a force during an exercise, in order to 


develop standard operating procedures and tactics, and to explore the limits 
of capability, etc. 


¢ evaluation of the maintainability and stores support for equipments and 
systems, to determine whether or not the complement, stores allowances, 
documentation, etc. are adequate. 


Typically, the Follow-on T&E program will last from 12-24 months, 
depending on the class of ship. At the end of this period RANTAU compiles a report 
in consultation with Maritime Headquarters Staff, the ship and other authorities as 


appropriate, indicating: 


e The results of the trials and evaluations 


e Whether any changes are recommended to the proposals by the PD for 
correcting deficiencies which were extant at the end of PAT&E Part 2 trials. 


¢ What additional items, if any should be added to the list of deficiencies and 
what action is proposed to overcome them. 


The AB assesses the results and provides the overview to ensure the 
program is comprehensive. This assessment serves to confirm or modify the 


assessment at AINS. 


G. RAN OT&E PROCEDURES 


Procedures are the methodology by which policy is carried out. This section 


discusses the procedures adopted by the RAN in implementing OT&E policy. 
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1. User Requirement to the Acceptance Management Plan. 


The development of the user requirement, and its subsequent progression 
through to the OT&E test documentation (Figure 15) is through the following 


documents: 


e Operational Concept Paper (OCP) 

¢ Defence Force Capability Proposal (DFCP) 
¢ Required Operational Characteristics (ROC) 
¢ Approved Ship Characteristics(ASC) 

¢ Test and Evaluation Master Plan (TEMP) 

e Acceptance Management Plan (AMP) 


FULLSCALE 
PROGRAM ENGINEERING PRODUCTION PRELIM  AINS OA 
INITIATION DEVELOPMENT = APPROVAL 

APPROVAL 


Bc stk ATION| FULLSCALE 
CONCEP TU ENGINEERING | PRODUCTION INSERVICE 
EVALU ATION DEVELOPMENT 





Figure 15: Progressive Definition of OT&E Documentation 
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a. Operational Concept Paper (OCP) 

The strategic basis of papers (DOA-87 [Ref. 6] and ASP-90 [Ref. 5]) 
provide endorsed Government policy guidance for force structure planning. These 
documents identify ADF roles and establish broad priorities for capability 
development. From this guidance, Operational Concept Papers are produced by 


HQAPF to identify the way ahead for specific force structure issues. [Ref. 54:p. A-4] 


b. Defence Force Capability Proposal (DFCP) 
Against the background of an OCP, a Defence Force Capability 
Proposal (DFCP) is developed by HQADF. The DFCP is the document against 
which specifications are produced and ultimately the proposed solution is evaluated 


as being suitable for service use. [Ref. 54:p. A-5] 


c. Required Operational Characteristics (ROC) 


To meet the overall requirements of the DFCP, more specific 


requirements may be described in the Required Operational Characteristics. 


d. Agreed Ship Characteristics (ASC) 
More detailed requirements, usually for ship programs, are developed 
into the Agreed Ship Characteristics (sometimes termed Approved Ships 
Characteristics). Other subordinate performance documentation may be developed 


also. 


e. Test and Evaluation Master Plan (TEMP) 


For a new ship design, the standards against which the ship is assessed 
are those derived from the then approved DFCP, ship characteristics and policies. 
These standards are developed into a TEMP which details the responsibilities for the 
specification of requirements, objectives, criteria and conduct of T&E, together with 


the schedule of activities, resources and key contractual dates. The most important 
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part of the TEMP is the statement of Critical Operational [ssues (COI) which are 
either of high technical risk or vital to the continuing development process. These 
issues may not be dependent on thresholds, but should cover all areas that affect the 
systems capability to accomplish its mission in a combat related environment. [Ref. 
44:p. 5] 

jf. Acceptance Management Plan (AMP) 


The Trials Authority develops management plans for the trials it 
conducts. Where those trials lead to AINS, the management plan is termed the 
Acceptance Management Plan. The AMP takes the COIs from the TEMP and details 
the tests and trials necessary to evaluate them to support an AINS decision. 
Similarly a Follow on OT&E (FOTE) Management Plan may be produced detailing 
the trials for the OT-4 phase. The detailed trials plans are developed from these 


management plans. 


g. Summary 
As in the USN, the hierarchy of requirements documentation in the 
RAN provides the linkage from capability requirement to operational test program 


necessary to evaluate the final product against the user requirement. 


2. Assessment of Operational Effectiveness 


The RAN has found it difficult to define exactly what is required to assess 
operational effectiveness. The RANTAU guidance on the conduct of OT&E [Ref. 
53] recognises that the user requirement is the bench mark for determining the 
degree to which a product is effective. Also that operational effectiveness is best 
assessed by a performance demonstration by normal operating personnel in the 
normal or given environment. But otherwise provides very little guidance on the 


assessment of operational effectiveness. Techniques employed involve analysing 


105 


each COI and employing modified Ship Qualification Trial and Fleet Exercise 
Program techniques to evaluate them. Use has also been made of simulation, 
weapon analysis and exercise analysis. The lack of suitable range facilities has also 
limited the assessment of operational effectiveness. 

This lack of operational effectiveness assessment capability in the RAN is 
due to most systems being acquired from large parent Navies. Consequently, the 
RAN has seldom had the need to assess operational effectiveness for itself. The 
Submarine Project is acquiring a variety of ranges for DT&E applications, e.g., the 
measurement of signatures, manoeuvreing characteristics, weapon control 


characteristics, etc. These ranges may also be suitable to support OT&E. 


3. Assessment of Operational Suitability 


The assessment of operational suitability in the RAN has been 
controversial, particularly the assessment of supportability. This quote from the then 
Director of Naval Integrated Logistic Support Management [Ref. 40] in 1987 
highlights a particular viewpoint: 

While there are clearly visible yardsticks by which production or weapon 
system performance can be measured, the success of logistic planning is to an 
extent invisible. Shortcomings in the assessment and provision of support will 
only become evident due to operational failure. All that RANTAU or the Ship 
Acceptance Board can do is make a subjective judgement as to whether all 
necessary ILS elements have been addressed and ensure that an acceptable 
follow on logistic support and enhancement process has been set in place. 

Before 1987, the assessment of ILS, a subset of operational suitability, was 
rather subjective. The assessment of supportability, for example, consisted of a 
review of allowance lists by experienced technical specialists where potential 


deficiencies were highlighted. The ILS community were concerned that the onboard 


spares allowances developed by detailed logistic support analysis, were being 
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subject to a relatively simplistic, qualitative assessment. Similarly the assessment of 
Reliability Maintainability and Availability (RM&A) was superficial with only 
those defects noted during the operational assessment period (from five to seven 
weeks) being analysed. A new approach was taken in 1987 to the assessment of 
operational suitability, and to ILS in particular. The assessment process has 


continued to be refined with the areas in Table 11 now assessed: 


TABLE 11: SUITABILITY ISSUES FOR ASSESSMENT 


Suitability Issues 


Availability 

Reliability 

Maintainability 

Safety 

Human Factors 

Interoperability 

Compatibility 

Integrated Logistic Support including: 
Maintenance Planning 
Supply Support 
Support and Test Equipment 
Technical Data and Documentation 
Manpower and Personnel 
Training and Training Devices 
Facilities 
Packaging, Handling, Storage and Transport 
Computer Support 
Configuration Control 





a. Suitability Assessment Procedure 


Safety is assessed by inspection and observation during system 


operation. Human factors, interoperability and compatibility are assessed by 
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observation during operational effectiveness testing. The assessment of ILS is 


conducted in two phases: 


(1) ILS Structural Review. Conducted prior to operational testing, this 
review assesses if the ILS intended to provided will support the ILS policy of the 
Project. It consists primarily of a review of the ILS Plan and discussions with the 


functional areas handling the support. 


(2) ILS Operational Review. During the operational testing period, the 
ship’s ILS related documentation and records are reviewed, maintainability 
demonstrations are performed and a supportability assessment is conducted. The 
supportability assessment still consists of a review of the allowance lists by 
experienced technicians, but now it also includes an analysis of spares usage and 
availability, from the time records are commenced to the end of the operational 
testing. 

RM&A is assessed by the collection and analysis of data during the OT- 
3 phase of operational testing to give an indication of performance prior to AINS, 
and then continued for 12 months during the OT-4 phase. The RAN is currently 
developing a routine in-service RM&A data collection and analysis system, 
however, until that is operational, the data is collected and analysed on an ‘as 


required’ basis by the Trials Authority. 


4. OT&E Reporting 


RANTAU coordinates the issue of trials reports which are issued on 
completion of each segment of OT&E e.g., o/c ASW phase of OT-3 etc. A final trials 


report is issued to the AB for inclusion in their AINS report. 
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5. OT&E Coordination 
A T&E Planning Group is formed prior to TEMP development under the 
leadership of the T&E manager for the project. Foundation members of the TEPG 
are the Project Director, Project Sponsor, the Design Approval Authority and 
RANTAU, with other parties seconded/called upon as required. [Ref. 44:p. 5] The 
TEPG analyses T&E requirements and estimates resources required to meet the 


requirements. 


6. OT&E Personnel Selection and Training 

Although OT&E is a specialised discipline with its own philosophy and 
methodology, the RAN has no training courses on OT&E, or on T&E in general. A 
number of officers have completed the USN Operational Test Directors Course in 
recent years which has improved the knowledge and understanding of OT&E within 
CTE’s organisation considerably. These courses have been arranged on an ad-hoc 
basis through overseas visit submissions rather than as pre-requisite courses for 
particular billets. No special selection criteria is involved for posting OT&E 
personnel or project T&E managers. Although some personnel may develop into 


OT&E specialists, no career path is consciously provided. 


H. RAN OT&E IMPLEMENTATION 
The implementation of OT&E policy within the RAN maybe demonstrated by 


reviewing examples of ship OT&E conducted to date, and that planned for ships due 


to enter service in the near future. 


1. DDG Modernisation Project 
The RAN has three DDGs built in the USA to a modified “Charles F. 


Adams” (DDG-?2 class) design. These ships underwent a major modernisation and 


refit over the period 1987-1991. The major modernisation occurred with the ships’ 
combat system, command and control system, sensors and weapons, with the aim of 
increasing the ‘supportability’ of the systems rather than increasing the ships’ 
capability [Ref. 39]. The upgrade was similar to that undertaken on a small number 
of USN DDGs. Most of the systems were imported and installed locally, however, 
the combat system operational program was developed in Australia. The nsk of the 
project failing to meet operational requirements was low, but the RAN needed to 
quantify the operational effectiveness and operational suitability of the modernised 
DDG. 

OT&E consisted of an OT-3 assessment during a normal SQT period, prior 
to workup and the Operational Readiness Evaluation. This was followed by an OT- 
4 phase, termed a Combat System Evaluation (CSE), for the first of class. This OT- 
4 was event based and was completed over a 12 month period, integrated with the 
normal ships program. Procedures developed during the OT&E for the DDG 
Modernisation included the use of the Combat System Trainer (CST) van for 
simulating complex AAW engagements, the use of system modelling and analysis 
by the Weapons Systems Research Laboratory (WSRL), and the use of new ILS and 
RM&A data collection and assessment procedures. 

DDG Modernisation was a low operational risk project. The OT&E 
followed the established RAN policy, however, it was the first to actually implement 
an OT-4 period to more fully assess operational effectiveness and operational 


suitability leading to OA. 


2. Australian Frigate Project 


Two US FFG-7 class frigates, known as Australian Frigates (AF), are being 
built in Australia. The frigates, FFG-06 and FFG-07, are being built to the HMAS 
DARWIN FFG-04 (USN FFG-44) configuration baseline except that the AN/SQS- 


IAS) 


56 sonar is replaced with the Australian Mulloka sonar. Changes to the confi guration 
were kept to a minimum commiserate with safety and operational improvements 
approved for incorporation into FFG-04.[Ref. 51:p. ix] All the identified equipment/ 
system differences between HMAS DARWIN and the AFs had undergone T&E 
prior to installation in FFG-05. 

As the AFs have an almost identical configuration to the last FFG-7 class 
purchased from the U.S., the major risk was in the production, not in the operational 
effectiveness and suitability of the finished product, hence OT&E was limited to an 
OT-3 period. The objective of OT&E for the AFs is to ensure that the user 
requirements have been met, and to provide an estimate of operational effectiveness 
and suitability in the RAN operating environment prior to release of the ship to 
operational service. The OT&E comprises an OT-3 period, which assesses the AF 
against the baseline established by HMAS DARWIN, including any approved 
changes. [Ref. 51:p. 10] The scope of OT-3 includes a Light Off Examination, Test 
Procedures, CSSQT and Post PSA Trials. OT-3 is being conducted on each ship and 


involves: 


e Installation Inspections 

¢ Harbour Phase System Qualification Trials 
e Sea Phase System Qualification Trials 

¢ Communications System Operability Trials 
e Integrated Logistic Support Assessments 

e Fleet Inspections 

e Air Certification 

e Light Off Examination 
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As the significant operational difference in the AFs is the Mulloka sonar, an 
Operational Performance Demonstration (OPD) was planned to determine if the AF 
can fulfil it's ASW roles and functions with the Mulloka sonar. One week of trials 
was planned during the OT-3 period to ensure that the Mulloka sonar system had 
been correctly installed and met its operational performance characteristics. This 
period was found to be insufficient, primarily due to technical problems, and a 
“mini-OPEVAL” is now planned to quantify the system’s performance. [Ref. 61:p. 
A-2] The acceptance schedule (Figure 16) follows established RAN policy. 


ACTIVITIES 


PAT-4/5 
Trials 


MILESTONES 


Delivery / 
Preliminary 
Acceptance 


Acceptance into 
Naval Service 





Figure 16: Australian Frigate Acceptance Schedule 


The TEMP, governing all tests, trials, and evaluations up to AINS, was 
produced as an Australian Frigate Addendum to the U.S. FFG-7 Class TEMP. A Post 
Delivery Test and Trials Plan [Ref. 52] was produced to serve as an advance 


planning document and an implementation plan. 


lil 


An Acceptance Board was established, initially for FFG-05 only. The 
PDT&T Plan includes the requirement that the AB is responsible for providing the 
Final Contract Trials agenda and conducting the trials[Ref. 52:p. 3-17], which is not 
in accordance with RAN AB policy. In its report at AINS, the AB considered that the 
application of the AB process was worthwhile and recommended that it should be 
followed for FFG-06, which is contrary to RAN policy of establishing the Board for 
first of class only. The Board also noted that acceptance documentation was 
contradictory and unclear. [Ref. 61:p. 2] 

The AF is a low operational risk project, except for the addition of the 
Mulloka sonar. The OT&E followed the established RAN policy, however, it failed 
to plan an OT-4 period to more fully assess operational effectiveness and operational 
suitability of the Mulloka sonar. An OPEVAL to fully assess the performance of the 


sonar has now been included. 


3. ANZAC Ship Project 
Australia and New Zealand are cooperating in the building of the ANZAC 
frigate. Essentially a Blohm and Voss MEKO 200 design incorporating a 
combination of European and US equipment, the design is classified as low risk, 
although modifications to the design were necessary to meet Australian 


requirements and to facilitate construction in Australia. [Ref. 58] 
a. ANZAC Ship OT&E 
OT&E for the ANZAC ships is currently planned to include: 


(1) OT-2 during which RANTAU may witness specific contractor 
testing, and elect to conduct initial operational evaluation using the facilities 
provided as part of the Combat System Support Centre, a LBTS established initially 


for the conduct of DT &E, and then to provide in-service system support. 
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(2) OT-3 consisting of a series of trials conducted to provide an initial 
assessment of the operational effectiveness and suitability of the ANZAC ship 
against the user requirement culminating in AINS. OT-3 will test the critical 
operational issues and will involve sea trials in an operational environment and a 


Ship Qualification Trial (SQT). OT-3 will be conducted on each ship. 


(3) OT-4 consisting of a follow-on series of trials conducted to further 
assess the operational characteristics of the ANZAC ships, particularly as an 
integrated unit in ASW, ASUW, AAW and utility roles. This will be conducted on 
the first of class only and includes participation in two major exercises with multi- 
threat environments, and appropriate freeplay.[Ref. 57] Both OT-3 & 4 will also 


involve an RM&A assessment over a two year period. 


Although only in the early stages, the planned OT&E for the ANZAC ship 
follows the established RAN policy, with the OT-3 period combined with the SQT, 
being conducted prior to AINS. The ship class has a higher operational risk than 
either the modernised DDG or the AF. 


4. COLLINS Class Submarine Project 


In 1987 the Swedish Kockums Type 471 design was selected to meet the 
requirements for the RAN’s new submarine. The six boats will be the first 
submarines to be constructed in Australia and will have a unique equipment fit. 
Work on Submarine 01 began in February 1990, with the launch planned for 
1993.[Ref. 60] Post delivery T&E activities, proposed by CTE, to be conducted by 
the RAN are outlined in Table 12. 
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TABLE 12: COLLINS CLASS POST-DELIVERY TRIALS [Ref. 59] 


Bech Contract). Contract) lie i SS samme 
a ee _ 
Sn 







a. CAT 6 Testing 


CAT 6 testing consists of Safety and Operational Work-ups, First of 
Class Trials, Operational Certification Trials, and OPEVAL. 


(1) First of Class Trials (DT-3). These trials will establish the actual 
performance of the COLLINS design and the in-service safe operating limits and 


conditions for the class. 


(2) PAT-6. These trials will be the Operational Certification Trials 
conducted after delivery in support of AINS of the particular submarine. The criteria 
will be based on applicable thresholds in the Agreed Ship Characteristics and the 
results of First of Class Trials with submarine 01. The principal events in PAT 6 


include: 


e Safety and operational work-up 

e Manoeuvreing trials 

e Weapon handling and discharge 

¢ Combat system operation, including weapon firings 


e Signature measurements 


Halls 


(3) OT-3. These trials are planned to evaluate the first of class only, 
after delivery, conducted in order that, in conjunction with the results of PAT&E 
Part 2, a recommendation for AINS may be made. OT-3 includes an ILS and RM&A 
assessment, and an OPEVAL of four to six weeks during which the submarine will 
be tasked to conduct a series of operations representative of the mission profiles 
against simulated threats and targets of interest. These trials may be reduced after 
experience with the first submarine. 


The proposed CAT 6 schedule is detailed in Figure 17. 
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Figure 17: HMAS COLLINS - Proposed CAT 6 Schedule (Ref. 49] 
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Of note is that the OPEVAL is planned to be conducted following the 
completion of DT-3, work up and the Operational Readiness Evaluation (ORE). So 
it will be conducted on a materially proven and operationally worked up submarine. 

b. CAT 7 Testing 


CAT 7 testing, comprising OT-4 exclusively, will consist of: 


(1) Long Term RM&A and ILS Evaluation. This will determine with 





greater statistical confidence, the Reliability, Maintainability and Availability of the 


submarine class, and assess the implementation of ILS items. 


(2) Measures of Effectiveness (MOEs) Testing. Some MOEs will 


require determination by statistically significant numbers of tests. However, to 
minimise resource requirements the tests will be constructed to allow the total test 
time and asset requirements to be minimised. To accommodate other scheduling 


limitations, this phase will probably occur in a number of stages. 


(3) Major Exercise. This phase assesses the performance of the 


submarine in the roles required of it during a major exercise. 


(4) Operational Deployment. This phase evaluates the performance of 
the submarine and the support infrastructure during an operational deployment and 
maintenance period in a foreign port. 


The proposed CAT 7 schedule is detailed in Figure 18. 
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Figure 18: HMAS COLLINS - Proposed CAT 7 Schedule [Ref. 49] 


c. Use of Land Based Test Sites 


The submarine project includes the construction of a combat system 
simulator and propulsion system simulator, both for system development and 
training applications. Some OT&E activities may be conducted by CTE on these 
systems at the land based test sties on a non-interfering basis to the contractor. [Ref. 


59:p. 20] 


d. Collins Class Submarine OT&E Summary 
Funding provision for RAN T&E up until AINS has not been included 


in the overall financial budgeting for the New Submarine Project [Ref. 59:p. ii]. To 
assist in the progressive development of the TEMP and to plan and agree those 


activities that will be required for AINS and OA, a TEPG has been established. The 
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detailed planning, preparation, execution and reporting of RAN T&E activities is the 
responsibility of a subordinate T&E Working Group. 

The Collins class submarine possibly has the highest operational risk of 
all current major RAN combatant projects. Consequently the increased need for 
OT&E has been recognised by CTE. The planned OT&E program demonstrates a 
departure from the usual RAN OT&E policy, and reflects a growing awareness 
within CTE of the need to separate DT and OT activities, and to include a dedicated 
OT-3 period after the usual SQT and ORE. 


5. OT&E Implementation Summary 
These examples of RAN implementation of ship OT&E policy illustrates a 


number of key issues: 


e The operational risk of ship projects within the RAN is increasing as the new 
RAN combatants have a unique weapons systems configuration. 


¢ OT&E is not generally conducted to support the production or purchase 
decision of ships. 


¢ The RAN relies on the results of a combined SQT and OT-3 period to provide 
a basis for AINS. 


¢ ILS and RM&A assessments are becoming more effective and credible. 
¢ Modeling and simulation is increasingly being used to assist OT&E. 
¢ Funding of ship OT&E is not being included in original project estimates. 


e Acceptance and T&E documentation is contradictory and unclear. 


I. RAN OT&E ISSUES 


1. What is OT&E? 


Although the term “OT&E” in the RAN is defined, and the hierarchy of 
Defence and RAN documentation addresses OT&E, the policy for the conduct of 
OT&E in the RAN is not specific and is included with general T&E policy. 
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Combined with a lack of distinction between DT-3, PAT-6 and OT-3, this leads to the 


few specific OT&E policy statements often being contradictory and unclear. 


a. Independence of the OT&E Authority 
OT&E is specified in RAN T&E policy [Ref. 44:p. 3] as being 


conducted by an authority independent of the developing and production agencies, 
and the results of OT-3 support an AINS decision. However, CPO Manual [Ref. 
38:p. 14-7] states that the Design Approval Authority (part of the developing 
agency) offers support to trials establishments during OT&E in order to provide 
advice to the Project Director on the suitability of the product for Acceptance into 
Service. This leads to conflicts between the DAA and the Trials Authority during 
the conduct of OT&E. 


b. Confusion between DT-3, PAT-6 and OT-3 
RAN T&E policy [Ref. 44:pp. 2-3] explains the T&E phases in the 


procurement process and then includes a note stating that OT-3 and OT-4 will 
incorporate the requirements of PAT-6 and PAT-7 Inspection Tests and Trials as 
described in ABR 1921(Reference 41). ABR 1921 however, defines many tests as 
being PAT&E, that RANTAU [Ref. 43] define as OT&E. Also DT-3 is shown as 
being conducted during the same period, however, the philosophy, aims, objectives 
and methodology of DT&E, PAT&E and OT&E are totally different. Perhaps this 
overlap or “blurring” of definitions arose because RANTAU was not only the RAN 
OT&E authority, but also the RAN T&E authority who conducted aspects of DT&E 
and PAT&E in addition to OT&E. With a large part of its workload devoted to 
conducting trials following the routine refit of combatants, RANTAU conducted 
HATS and SATS which were understood to be PAT&E and then went on to conduct 
the more operational Ship Qualification Trial (SQT) which was perceived to be 


OT&E. The SQT period, however, is rarely a free play environment, makes limited 
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use of countermeasures, and is conducted prior to the crew being fully worked up. 
Also contractors are often still onboard making final adjustments to systems and 
assisting the crew with operation and/or maintenance. The SQT is essentially a 
PAT&E function which may include some components of the operational 
effectiveness and suitability aspects of OT&E. 

The problem with this “blurring” of definitions goes beyond the 
semantics. The results of OT-3 are used to support an AINS decision, but as it is 
really PAT&E, it assesses more the ability of the ship to meet technical or Fleet 
standards. It does not truly assess the operational effectiveness and operational 
suitability of a new ship against the user requirement. Therefore, a ship is not truly 
subject to OT&E against the user requirement before it is granted AINS. 

This lack of distinction between T&E types also confuses Project 
Directors and other Defence managers. It leads to a lack of consideration of the 
importance of OT&E, and hence the need to consider OT&E funding early in a 
Project. It also leads to confusion over T&E management responsibilities. It 
confuses the role of the Acceptance Board, which is really assuming a quality 
control function over what is essentially PAT&E testing, rather than it’s members 
using their knowledge and experience in assessing the results of true OT&E against 
the user requirement. 

The difference in terminology used by different documents and 
authorities also creates confusion. For example, the term OPEVAL is defined [Ref. 
38:p. 14A-2], but terms such as mini-OPEVAL [Ref. 61], Operational Performance 
Demonstration [Ref. 51] and (Ship) Operational Trials [Ref. 41][Ref. 44] are not. 


2. OT®&E and Acceptance of New Combatants 


With the construction of new frigates and submarines, the RAN has 


embarked on its greatest shipbuilding program since the Second World War. 


Although these combatants are being built to proven designs and, therefore, are 
considered relatively low risk, they do present a greater operational risk than recent 
RAN combatant programs. In September 1991, CNS expressed concern that the 
RAN “may not be adequately placed” to properly manage the comprehensive and 
complex procedures necessary to accept the ANZAC ships and Collins class 
submarines into service [Ref. 62]. This has focussed attention on the acceptance 


procedures and consequently on the OT&E of the new combatants. 


3. The New RAN OT&E Organisation 


RANTAU ceased to exist as an organisation on May 15, 1992. The rationale 
leading to this decision is important as it is indicative of the level of understanding 


of OT&E within the RAN. 


a. Background 


RANTAU was established in 1966, and originally administered by 
Navy, but later by the Defence Scientific and Technical organisation. In 1982 
control of RANTAU was returned to Navy with the stated purpose of conducting 
operational testing and evaluation, conducting inspections and providing 
operational assessment data.[Ref. 46:p. 1] Functional and operational responsibility 
for RANTAU was transferred from the DCNS Division of Navy Office to the 
Maritime Commander on 31 July 1991 concurrent with the introduction of the 
Program Management and Budgeting System(PMBS). The Director of RANTAU 
was made responsible to the Maritime Commander for the management of 


RANTAU. 


b. What Did RANTAU Achieve? 


As a result of this change in the organisational structure (a change due 


fundamentally to financial funding and accounting requirements) there was a need 
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to question what RANTAU achieved overall, and whether its tasking should be 
amended to more accurately reflect the needs of the Maritime Commander in 
maintaining, and where necessary improving, the operational capability of the 
Maritime Force. A study was directed [Ref. 46] to: 


e focus on the tasks required to be undertaken to assess the operational 
effectiveness of Maritime Force units and the Force as a whole. 


e study any aspects of RANTAU’s current employment which were 
inappropriate given its functional and operational responsibilities to the 
Maritime Command. 


The results of this study [Ref. 47] found that only about 20% of the 
work carried out by the RANTAU Tnials Unit was carried out for the Maritime 
Commander. This consisted of SQTs, and HATS and SATS. The large majority 
(80%) of the work done by the Trials Unit was found to be done on behalf of the 
Assistant Chief of Naval Staff - Materiel (ACMAT-N) in accordance with ABR 
1921(Reference 41). The report stated that the work conducted for ACMAT-N 


included: 


¢ assisting with the wnting of the TEMP for every project. 

e installation inspections as part of PAT&E Part 1. 

¢ heavy involvement with PAT&E Part 2. 

e the conduct of OT&E for most projects. 

¢ support for the post delivery acceptance into Naval Service (AINS) 


process.[Ref. 47:p. 17]. 
c. Organisation Options 
The report proposed transferring the Trials Unit, less a proportion to 
support Fleet Trials, to the Materiel Division. The unit would then be ideally situated 
to form the nucleus of an organisation with responsibility for the coordination and 


administration of all trials conducted for and/or by the RAN. It could also form the 
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core of the AB, with perhaps the Director being the standing vice president of the 
body. 

The Materiel Division rebutted this argument with the perception that 
an independent audit authority is necessary to maintain production standards. It 
appears that there is a belief that if the RAN’s trials authority was to be a functional 
unit of the Materiel Division, then the head of the Division would be able to 
influence trials results. The report contends that this argument appears to point toa 
management problem rather than a structural issue, but in any case it did nothing to 
counter the argument that the RANTAU Tnals Unit is not an appropriate component 
of the Operations Sub-program of the PMBS (i.e., the Maritime Command). A 
solution to this problem would be to return the Trials Unit to the Executive Sub- 
program (i.e., Navy Office). However, it recognised that the service provided by the 
trials authority was also largely unrelated to that sub-program. The report then 
considered the impact of HQADF, who occasionally tasks RANTAU in support of 
projects for the development of the maritime component of the ADF. 

The report recognised that this seemingly simple organisation of 
RANTAU was, in fact, complex due to the variety of tasks imposed upon the unit 
by a number of masters on a day to day basis. It concluded that as a single unit, 
RANTAU does not rest comfortably with the current arrangement and nor does it 


with any other Navy Sub-program for budgeting and management purposes. 


d. An “Audit” Function 


The report states that RANTAU served an apparent independent audit 
function for the Materiel Command, and that: 
this function was something that had evolved over time rather than in response 
to any specific need. It was beyond the scope of the original purpose for the 


organisation which became RANTAU, and the absolute need for such an audit 
service in the era of PMBS and tight fiscal control is questioned. An audit 
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service for the Materiel Division does not seem an appropriate sub-component 
for the Combat Forces sub-program. 

The report concluded that, other than for a small trials unit to conduct 
trials in support of units assigned to the fleet, it is inappropriate for the Maritime 
Commander to have functional and administrative control of a Trials Unit which 
largely serves the Materiel Division. The options appeared to be the transfer of the 


Trials Unit to the either the DCNS or Materiel Divisions. 


e. The OT&E Function 
The report, however, failed to recognise that RANTAU was the OT&E 


authority. As such, the “audit” work apparently done for the Materiel Command was 
really done on behalf of the sponsor (i.e., HQADF) and the user (i.e., the Maritime 
Command), as OT&E is the final feedback that the item produced by the Materiel 


Command meets the user requirement. 


f. Current Trials Organisation 
Following the release of this report, and due to the RAN’s decision not 
to adequately staff and fund RANTAU as an independent unit, the functional tasks 
of RANTAU were incorporated within the framework of the Maritime Headquarters 
[Ref. 48]. The current placement of the former Trials Unit, now known as 
Commander Test and Evaluation, within the Maritime Command is shown in Figure 
19. Of note is that the OT&E authority is now responsible to the Maritime 


Commander through the in-service engineering management chain of command. 
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Figure 19: Commander Test and Evaluation Organisation 





J. SUMMARY 


1. Importance of OT&E 


The importance of T&E i general is recognised in the acquisition process, 
to identify problem areas, to allow timely corrective action to be taken, and to reduce 
technical risk. As Australia has a policy of procuring low risk, operationally proven 
systems, which may be constructed locally, rather than higher risk local 
development, the RAN concentrates on post delivery testing, rather than testing to 
support a production or purchase decision. The importance of OT&E to the 


acquisition process, however, is not adequately appreciated. The RAN appears to 
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have its focus on PAT&E to meet technical and fleet standards, rather than an 
assessing operational effectiveness and operational suitability against the user 


requirement. 


2. OT&E Policy 
The lack of emphasis given to OT&E within the RAN is possibly due to the 


lack of clear guidance on OT&E in the T&E policy documentation. The decision to 
subject an acquisition to T&E will initially be decided by the Sponsor and detailed 
in the Capability Proposal, however, the Sponsor and other acquisition managers 
require clear guidance on T&E to enable them to make effective decisions. The 
categories of ship T&E used in the Australian DoD have been derived from the U.S. 
DoD instructions, but interpreted to accord with Australian Requirements. During 
this interpretation there has been a “blurring” of the types of T&E, resulting in a lack 
of clear delineation between the T&E types. This problem was further compounded 
by the adoption of the PAT&E based, USN Total Ship Test Program system, and 
modifying it to include DT&E and OT&E components. 


3. RAN OT&E Organisation 
The RAN and DoD have reorganised over the past five years to create a 
more effective and efficient user requirements organisation, and to better accord 
with Program Management and Budgeting principles. However, as a result, the 
RAN has no T&E, or OT&E policy maker and overseer, no OT&E coordinator 
within Navy Office, and the RAN’s OT&E authority is buried within the in-service 


engineering management area of the Maritime Command. 


4. Understanding of OT&E 


As a result of incomplete, contradictory and confusing T&E policy 


documentation, emphasis on PAT&E and the “blurring” of T&E types, the lack of 
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true OT&E actually conducted and few personnel experienced in OT&E, the 
understanding of OT&E within the RAN is poor. As a consequence, the philosophy, 
methodology. and benefits to the RAN of OT&E are not adequatel y recognised. 


5. Summary 


The demise of the RAN’s independent OT&E authority, RANTAU. and its 
subsequent integration within Maritime Headquarters as the Commander, Test and 
Evaluation (CTE), was the latest in a series of reorganisations that has significantly 
reduced the capability and authority of the OT&E community. The new combatants 
currently under construction for the RAN, although relatively low risk by USN 
standards, are a higher operational risk than previous RAN ship programs. Although 
using proven technologies, both the ANZAC ships and Collins class submarines 
have unique weapon and sensor fits, not currently in service with large parent navies. 
The conduct of PAT&E only, is not considered sufficient to measure the operational 
effectiveness and suitability of these new combatants to determine, with confidence, 
that they meet the user requirement. This can be achieved by the conduct of an 


effective OT&E program. 
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V. USN/ RAN OT&E COMPARISON 


This chapter compares the characteristics of the USN and RAN OT&E systems 


and highlights significant differences. 


A. DEFENCE ACQUISITION PROCESS 


The Defence system acquisition life cycle of both nations consists of 
progressive development phases separated by major decision milestones. 
Milestones are periodic formal program reviews for authorisation to advance to the 
next stage. The two systems are compared in Figure 20. With a policy to maintain 
world leadership in key Defence technologies, the U.S. places priority on 
development programs. The major decision point is centred on the decision to 
proceed beyond Milestone IIIA - Low Rate Initial Production , when the large 
dollars in an acquisition program are due to be spent in the production phase. 
Passing a formal, independent OPEVAL is its major criterion for approval to Full 
Rate Production. 

Although the decision to produce or purchase a new system is important, the 
Australian DoD prefers lower risk, proven technologies for the majority of new 
Defence acquisitions. In many cases, systems with the fundamental capabilities 
required by the ADF are available overseas, however, some of these systems require 
additional development, adaptation or integration to meet Australian specific 
requirements. The Australian system, therefore, places a high priority on the post 
production phase with the three milestones of Preliminary Acceptance (PA), 
Acceptance into Naval Service (AINS) and Operational Acceptance (OA). 


However, Australia is now buying far less “off the shelf’ from overseas, and 
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systems are increasingly being designed and built in Australia, or at least adapted 


from overseas designs. 
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Figure 20: Comparison of U.S. and Australian Acquisition Processes 


B. T&E DEFINITIONS 


Definitions and terminology form the basis of a T&E system. As the categories 
of ship T&E used in the Australian DoD have been derived from the U.S. DoD 


instructions, both Navies recognise three types of T&E: 
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e Developmental Test and Evaluation (DT&E) 
¢ Production Acceptance Test and Evaluation (PAT&E) 
¢ Operational Test and Evaluation (OT&E) 


The categories within these types of T&E, however, have been interpreted by 
the RAN to accord with Australian requirements [Ref. 38:p. 14-1]. This 
interpretation leads to some subtle, but significant, differences in T&E definitions 


between the two Navies. 


1. Developmental Test and Evaluation (DT&E) 


DT&E is conducted to assist the engineering design and development 
process and to verify attainment of technical performance specifications and 
objectives [Ref. 11:p. 3]. Both Navies agree that DT &E is conducted in three major 
phases. Table 13 compares these DT&E phases. Minor wording differences aside, 
the three DT &E phases are essentially identical. The most significant difference is 


that the USN recognises PAT&E as a form of DT-3 testing. 


TABLE 13: COMPARISON OF DT&E PHASES 


USN Description RAN Description 


Demonstration that all technical nsk | DT-1 | Validation of design concept. 
areas have been identified & that best 

technical approaches have been 

accepted. 


Demonstrate that design meets | DT-2 | Proving design. 
specifications. 


Testing conducted on production items | DT-3 | Demonstration that production meets 
to ensure compliance with contracted required technical characteristics. 
specifications. PAT&E is a form of 

DT-3. 





it 


2. Production Acceptance Test and Evaluation (PAT&E) 


Both Navies agree that PAT&E is testing conducted on production items to 
ensure systems meet contract specifications and requirements. PAT&E for ships in 
the USN is managed under the Total Ship Test Program (TSTP) which includes 
seven phases of PAT&E. In recent years the RAN has adopted the TSTP, initially 
with the modernisation of the DDGs, and now in the testing of the locally produced 
FFG-7 class under the Australian Frigate Project. The adoption of the TSTP by the 
RAN included the redefinition of the test phases. A comparison of the PAT&E 
phases is provided in Table 14. There are major differences between the two Navies 
in these PAT&E categories. The USN system includes only PAT&E, whereas the 
RAN includes elements of DT&E and OT&E, in addition to PAT&E, and they are 


given PAT designations. 


TABLE 14: COMPARISON OF PAT&E PHASES 


USN Description PAT&E RAN Description 


Not defined in U.S. system PATO | Design and engineering development 
tests 

Matenal receipt inspection and shop | PAT-1 | Production and burn-in tests 

tests 

Shipboard installation inspections | PAT-2 | Environmental qualification tests 

and tests 


Equipment level operational tests PAT-3 | System development tests 


HIntersystemtests stem tests | PATS | Sea esting i Sea esting i 


en tests e.g., surveys, rangings ie Operational and Qualification Tnals 
(conducted with OT-3) 

Trials tests including sea trials, | PAT-7 | Follow on Operational T&E 

builders trials, acceptance tnals. (conducted with OT-4) 


132. 











3. Operational Test and Evaluation (OT&E) 


Both Navies concur that OT&E is conducted to determine a system's 
operational effectiveness and operational suitability, to identify system deficiencies 
and the need for potential modifications to meet established OT thresholds, and 


develop tactics. They also agree that OT&E has three distinguishing characteristics: 


e It is conducted in an operationally representative environment 


e [tis conducted on production representative equipment using fleet personnel 
for operation and maintenance. 


e Itis conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 


They both use the term “Operational Evaluation’, abbreviated as 
“OPEVAL” to cover T&E on production representative baseline equipment using 
the maintenance and support personnel and equipment for normal operational use 


which aims to: 


e Demonstrate operational effectiveness and suitability. 
e Provide data to assist in the development of tactical aspects of the equipment. 


e Verify data, handbooks and documentation covering the operation of the 
system. 


The USN uses the term OPEVAL as the final stage of OT-II, supporting a 
production decision. Although the Australian definition of OT-2 includes an 
OPEVAL, the term is not used exclusively for OT-2. It is used in many contexts 
from OT-2 to OT-4. 

There are also significant differences in the detailed OT&E definitions. The 
RAN includes ship Certification / Qualification trials in OT-3, whereas the USN 
categorises Certification trials as PAT&E and Ship Qualification Trials (SQT) as 
regular in-service PAT&E conducted by the INSURV Board and Type Commander, 
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not as OT&E. A comparison of the OT&E phases of the USN and RAN are given in 
Table 15. 


TABLE 15: COMPARISON OF OT&E PHASES 


USN Description RAN Description 


Early Operational Assessment OT-1 | Operational assessment of the 
development proposal 


OT&E conducted to support a| OT-2 | Demonstration of achievement of 
production decision. Final phase is program requirements for operational 
termed OPEVAL. effectiveness and suitability of a 
prototype (OPEVAL) to support 
proceeding to full production 






















































OT-3 | Demonstration of achievement of 
program requirements for operational 
effectiveness and suitability on 
production of ship/aircraft/system, 
normally in independent operations 
using normal Fleet personnel. 
Certification/Qualification Trials 
include limited  @ireliabilinae 
maintainability, availability and 
logistic supportability assessments. 


OT+4 | Demonstration of achievement of 
program requirements for operational 
effectiveness and operational 
Suitability on production of ship/ 
aircraft/system using normal Fleet 
personnel in a multi-force, multi-threat 
environment, Includes detailed 
reliability, maintainability, availability 


OT&E conducted on production 
system to verify correction of 
deficiencies after OPEVAL, and 
certification of operational 
effectiveness and suitability. 













Validation of the operational 
effectiveness and suitability of 
production systems. Usually in 
different environments and to assess 
integrated operation of system. For 
ship programs, is normally conducted 
to venfy that critical deficiencies have 
been identified and to complete any 
outstanding OT&E. 
















and logistic supportability 
assessments. 















Follow-on OT for assessment after 
modernisations, new applications or 
defect rectifications after OT-4 


Not defined in the U.S. System. OT-5 
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C. T&E IN THE ACQUISITION PROCESS 


Both Navies appreciate the importance of T&E in the acquisition process. T&E 
results are assessed as part of the milestone reviews. However, the level of 


importance of T&E, and OT&E in particular, differs greatly. 


1. The Requirement for T&E 


The importance of T&E, and OT&E in particular, to the progressive 
assessment of programs in the USN is such that its use has been mandated by the 
U.S. Congress and incorporated into the laws of the United States. The law includes 
the provision of a Director of Operational Test and Evaluation (DOT&E) in the 
Department of Defence. The law also addresses specific areas of OT&E reporting 
and conduct to ensure the Congress is kept informed, and the testing and reporting 
are impartial (Ref. 17]. By including the requirement that a major Defence 
acquisition program may not proceed beyond low-rate initial production until initial 
OT&E of the program is completed, Congress ensures that OT&E is an integral part 
of the acquisition process. 

The USN appreciates that, if adequate OT&E is not done, and the weapon 
system does not perform satisfactorily in the field, significant changes may be 
required. Moreover, the changes will not be limited to a few developmental models, 
but may also be applied to items already produced and deployed. The USN also 
recognises that, in extreme situations, it also risks deploying systems which cannot 
adequately perform significant portions of their missions, thus degrading its 
deterrent / defensive capabilities and endangering the safety of military personnel 
who operate and maintain the systems. 

The need for local T&E is recognised by the Australian Government and 
DoD, for both indigenous and overseas systems, however, the importance of OT&E 


in particular, is not specifically addressed in any Australian T&E policy document. 


Lys 


Perhaps this lack of recognition of the importance of OT&E is due to the RAN 
procuring low risk, complete ships in the past (e.g., FFG-7 class) with the wealth of 
USN DT&E and OT&E behind it. Also the RAN, unlike the USN, has not 
experienced the operational situations in recent years which may have highlighted 
operational effectiveness and suitability deficiencies in its ships. With the RAN now 
buying higher risk, unproven combinations of systems within its ships, it must now 


rely on its own OT&E. 


2. T&E Contributions At Major Milestones 


The USN OTE system forms an integral part of the acquisition process. 
Program advance from one phase to the next is not by the calendar of planned 
schedule, but by actual resolution of critical operational issues and achievement of 
pre-set thresholds verified by T&E [Ref. 11:p. 2]. OT&E in the RAN, on the other 
hand, may be conducted during each phase in the acquisition process, but it is not 
essential to a local system meeting a development or production milestone, or the 
local production or purchase decision of an overseas system. It is however, required 
before a system achieves Acceptance into Naval Service or Operational Acceptance. 
Figure 21 compares the contributions of T&E to the acquisition milestones for each 
Navy. 

Although the diagrams appear to be very similar, the RAN OT&E 
contributions to the acquisition milestones are by no means definitive. There is no 
documented requirement for the conduct of OT-1 and / or OT-2 prior to a production 
or purchase decision of major acquisitions. The decision to conduct OT&E being the 
responsibility of the Sponsor, Project Manager and the Defence acquisition 
committees. RAN T&E policy documentation almost exclusively focuses on the 


post production acceptance phases. Even during the acceptance phase, however, the 
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requirement for OT-3 is clouded in definition differences between policy documents. 


The RAN appears to have its focus on PAT&E. 
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Figure 21: Comparison of T&E Contributions to Acquisition Milestones 
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Of note is that the USN diagram shows separate activities for DI&E and 
OT&E related tests, while the RAN diagram groups them all together. This 
demonstrates the delineation between DT&E /PAT&E and OTE evident in the 
USN system. 


D. OT&E ORGANISATION 


The OT&E organisation consists of a number of functions. The authorities 


performing these functions in each Navy are listed in Table 16. 


TABLE 16: COMPARISON OF OT&E AUTHORITIES 


USN OT&E RAN 
Authority Function Authority 


Director, Policy Maker 
Operational Test & Evaluation & Overseer 
(DoD) 


N8 - Warfare Requirements Director General Force 

Chief of Naval Operations Development (Sea) 

Commander, Developing Assistant Chief of Naval Staff - 

Naval Sea Systems Command Agency Materiel 

Commander, Operational OT&E Tester | Commander, Test & Evaluation 

Test and Evaluation Force & Evaluator 

Ships Commanding Officers, User Ships Commanding Officers, 

Fleet Commanders Maritime Commander 
Director, OT&E 


Test, Evaluation & Coordinator 
Technology Requirements 
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1. OT&E Policy Maker and Overseer 


An OT&E policy maker and overseer is employed in the U.S. system 
responsible for policy formulation, evaluation and oversight of all OT&E. The 
Director of OT& E (DOT&E) is the principal OT&E official in the Department of 
Defence and the principal advisor to the Secretary of Defence on OT&E. In addition, 
a Director T&E, responsible for developmental testing policy reports, to the 
USD(A) in DoD. 

There is no clear OT&E, or T&E, policy maker or overseer within the 
Australian DoD or the RAN. The Chief of Capital Procurement [Ref. 38] addresses 
T&E in general, but more in the form of guidance rather than policy. The former 
Director General of Naval Warfare (DGNW) drafted the RAN Test and Evaluation 
Policy Defence Instruction [Ref. 44], however after the recent reorganisation, T&E 
policy for the RAN is presently in a state of flux, with no office having assumed the 


responsibility [Ref. 50]. 


2. Program Sponsor 


The program sponsor is responsible for the development of concepts for 
operations for naval warfare systems, and for acquisition program requirements and 
related system thresholds. He/She is also responsible for the subsequent trials and 
continuing overview to ensure that the equipment meets the requirement. Within the 
USN, major warship programs are sponsored by N8 - Warfare Requirements, under 
CNO. Under the new USN organisation, increasing importance is given to the Fleet 
and Type Commanders in determining requirements and allocating priorities. 
Besides generating the user requirement, the sponsor also has a review function in 
the OT&E process.[Ref. 20] 

The Australian Defence Force has a centralised force development and user 


requirements organisation. within HQADF, the Director General Force 


Nee) 


Development (Sea) having sponsorship of major RAN acquisitions. The decision to 
subject an acquisition to T&E in the RAN is initially decided by the sponsor and 


detailed in the capability proposal [Ref. 44:p. 1]. 


3. Developing Agency 

When a program is approved, a program office is formed to manage the 
acquisition of the system. In the case of the USN, this office is established within the 
Naval Sea Systems Command (NAVSEA). The program office is responsible for the 
conduct of DT&E in preparation for OT&E. To assist all NAVSEA programs with 
T&E matters, NAVSEA have established their own T&E of fice (SEA 60). Although 
primarily a DT&E policy authority, this office also has an input into the OT&E 
process by reviewing the TEMP, and by conducting OT&E readiness reviews of 
NAVSEA programs. 

Within the RAN, the Materiel Division is responsible for acquisition of the 
capital equipment to meet the requirement. Similar to the USN, the nature and extent 
of the T&E to be conducted is decided by the Project Director in consultation with 
the Design Approval Authonty and RANTAU. However, the Materiel Division does 


not have a T&E office to support Project Directors. 


4. OT&E Tester and Evaluator 
The Operational Test and Evaluation Force (OPTEVFOR) is the USN’s 


sole independent test agency responsible for the planning and conduct of OT&E. 
COMOPTEVFOR reports directly to the CNO, and is separate and distinct from the 
developing and procuring commands. 

The RAN Trials and Assessing Unit (RANTAU) was the RAN’s OT&E 
authority who also conducted DT&E / PAT&E on behalf of Project Directors and 
Ship Qualification Trials for the Maritime Commander. With the disestablishment of 


RANTAU in May 1992, these trials functions were transferred to the Commander, 
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Test and Evaluation (CTE) under the Chief Staff Officer (Engineering) in the 


Maritime Command. 


5. User 


The Commanding Officer and crew of the ship under test, and his Fleet and 
Type Commanders are the ultimate users of the system. The Maritime Commander, 
and the individual ships’ Commanding Officers and crews are the ultimate users of 
a ship. The personal assessment of the Commanding Officer of the ship under test is 
sought by OPTEVFOR during the conduct of OT&E [Ref. 15:p. 44]. The 
Commanding Officer of a ship under test usually provides CTE with his opinions on 


the ship’s performance at the conclusion of a trials period. 


6. Coordinator 


CNO has responsibility for ensuring the adequacy of the Navy’s overall test 
and evaluation program. T&E policy and guidance are exercised through the 
Director, Test, Evaluation and Technology Requirements (NO91). This organisation 
also acts as a T&E Focal Point and Coordinator, responsible for coordination of 
T&E matters in the designated Programs, System Commands and Department of the 
Navy. [Ref. 13:p. 7-5]He also chairs the Test and Evaluation Coordination Group 
The RAN currently has no authority with this role, however, the T&E manager 
within major projects is responsible for chairing the Test and Evaluation Planning 


Group. 


7. OT&E Organisation Summary 


A comparison of the OT&E organisation structures of both Navies is shown 
in Figure 22. The USN OT&E organisation is well structured with comprehensive 
responsibilities for each participant, and clearly delineated from the DT&E 


structure. COMOPTEVFOR is the truly independent tester and evaluator, being 


14] 


separate from developing, production and user agencies, and responsible only for 
OT&E. As a Rear Admiral and reporting directly to the CNO, he has authority, 
visibility and influence. The office of DOT&E provides a layer of oversight on the 
OT&E process to ensure objective testing is conducted, giving credible results and 
leading to impartial decisions. NAVSEA has its own T&E agency to assist PM’s in 
DT&E, and in preparation for OT&E. The OT&E process is coordinated from 
within CNO’s office by the Director, Test, Evaluation and Technology 
Requirements. 

Following a number of recent reorganisations, the RAN’s OT&E structure 
is in disarray. The RAN’s OT&E authority is now CTE within the Maritime 
Command. In addition to OT&E responsibilities, he also conducts some DT&E, and 
some functions equivalent to the USN INSURV board, OPTEVFOR and the Type 
Commander. Only of Commander rank, he lacks authonity, particularly with Project 
Directors and, being within the in-service engineering area, he possibly lacks 
credibility and influence on operational matters pertaining to OT&E. In developing 
programs, the RAN relies on the minimal interface between the centralised sponsor 
(DGFD(Sea)) and CTE to ensure appropriate OT&E is considered, since the sponsor 
has no agency within his own organisation to provide the necessary advice. Once a 
project is established, the Project Director performs the coordinating role through 
the TEPG. The Materiel Command, however, has no T&E agency to advise PD’s on 
this role, and to be the retainers of “lessons learned” from other projects. Finally, 
with no clear policy maker for T&E in general, or OT&E in particular, the RAN 
lacks the ability to effectively manage overall T&E, let alone OT&E, within the 
RAN. 
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E. OT&E POLICY 
USN OT&E policy has its basis in statutes approved by Congress. The RAN has 


no such formal basis and its OT&E policy is embedded in the more general T&E 


policy. A number of OT&E policies of interest include: 


1. OT&E Documentation 


USN policy is refined in the DoD acquisition Directives 5000. 1[Ref. 25] 
and 5000.2 {Ref. 24]. The essential T&E elements in these directives are further 
detailed in OPNAV Instruction 3960.10C [Ref. 11] and other subordinate 
instructions. USN OT&E documentation is comprehensive and provides clear, 
consistent and non-conflicting policies and guidance to OT&E participants. This 
concise documentation establishes the disciplined management approach to OT&E 
taken by the USN. 

The RAN does not enjoy this sound basis of OT&E policy. Some general 
guidance to assist Project Directors in assessing the need for T&E in a project and 
to provide guidelines for its conduct is provided in the Capital Equipment 
Procurement Manual (CEPMAN 1)[Ref. 38]. The top level T&E policy document 
for the RAN is the Defence Instruction (Navy) on RAN Test and Evaluation Policy 
[Ref. 44], while ABR 1921[Ref. 41], contains T&E and acceptance policy 
applicable to ships building, undergoing modernisation, conversion or extended 
refit. These documents provide broad T&E policy only and, other than defining 


OT&E, provide little policy as to its conduct. 


2. OT&E Requirements 


In the USN system, the requirement to conduct OT&E on new major 
acquisitions is mandatory. Although the Sponsor initially develops the system 


thresholds, the level and extent of OT&E required is delineated in the TEMP and 
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approved prior to Milestone I. The appropriate level of OT&E is determined by 
COMOPTEVFOR in consultation with the Program Manager, with DOT&E 
oversight. This ensures that OT&E planning is conducted and approved early so the 
costs of OT&E can be included in project funds. 

In the RAN, the decision to subject an acquisition to T&E is initially 
decided by the Sponsor and detailed in the Capability Proposal. The nature and 
extent of the T&E (no distinction is made in RAN policy between the various types 
of T&E) is decided by the Project Director in consultation with the Design Approval 
Authority and CTE. Any conflict over the requirement, nature and/or extent of the 
proposed T&E is resolved by DCNS and ACMAT-N for operational and design 
matters respectively.[Ref. 44:p. 2] The role of DCNS should now rest with 
DGFD(Sea) as the Sponsor. The determination of the OT&E required has 
traditionally not been achieved early enough in the life of a project, and so project 


funding has not included the provision for OT&E. 


3. Independence of OT&E Authority 


As OT&E is the final exam for an acquisition program, independence is 
necessary so that there is no question of impropriety. The USN requires its 
independent operational test activity to be separate and independent from the 
materiel-developing and procuring agency and the using agency. This ensures that 
the testing and reporting agency has nothing to gain or lose by whatever they say. As 
major acquisition decisions hinge on the results of OT&E within the USN, 
independence is necessary to ensure that the reports on which those decisions are 
based are not self serving.[Ref. 19] 

Although major acquisition decisions within the RAN are less based on 
OT&E, the RAN also maintains the policy that OT&E is to be conducted by an 


authority independent of the development and production agencies. With the demise 


145 


of RANTAU and the transfer of OT&E responsibilities to CTE on the staff of the 
Maritime Commander, independence from the development and production 
agencies is achieved, but not from the end user as in the USN. The implications of 
this lack of independence are that the end user may have his own aims and objectives 
which may conflict with the total impartiality of OT&E reporting. The politics of 
requirements and funding could come into play, as the Maritime Commander 
establishes scheduling priorities for his resources, and OT&E may take a low 
priority behind operations and training. The Maritime Commander may also not 
wish to “stress” the system, in case he perceives that operational deficiencies so 


highlighted may be seen as having some reflection on his own organisation. 


4. Contractor Involvement in OT&E 


To ensure impartiality and credibility of testing, the USN prohibits persons 
employed by the contractor for the system being tested being involved in OT&E, 
unless contractors are planned to be involved when the system is deployed in 
combat. 

The RAN requires that during OT&E, a system should be demonstrated 
“using normal Fleet Personnel”, however, the policy documentation makes no 
statement regarding contractor personnel involvement in system operation or 
maintenance during OT&E. It has been RAN practice (e.g., in the DDG 
Modernisation Project) to have contractors onboard resolving technical problems 
during the SQT period, which is included as part of OT-3 in the RAN, thereby 


possibly causing test bias. 


5. T&E Planning 
Both the USN and RAN use the Test and Evaluation Master Plan (TEMP) 


as the single, executive, long range planning document for T&E in a project. Both 


Navies are in agreement that approval of the TEMP constitutes establishment of a 
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contract between the requirement-setter and the PM which is essential to formal and 
accountable program execution, and between the requirement-setter and the OT 
agency, upon which the OT agency will independently evaluate system operational 
effectiveness and suitability. The main differences between the two Navies are in the 


responsibility and level of approval, and in the timing of the TEMP. 


a. TEMP Responsibility 

Within the USN, the TEMP is drafted by the PM. COMOPTEVFOR 
drafts Part Four on OT&E and provides OT&E resource requirements for Part Five. 
The TEMP is then co-submitted by both the Program Manager and 
COMOPTEVFOR. It is reviewed by the Assistant Chief of Naval Operations 
(Surface Warfare), the Director, T&E and Technology Requirements, and the 
Assistant Secretary of the Navy (Research, Development and Acquisition). It is 
ultimately approved at the DoD level by the Director, OT&E and the Director, Test 
and Evaluation. 

Within the RAN, the TEMP is the ultimate responsibility of the project 
sponsor [Ref. 44:p. 4]. However, during the period when there is a full time Project 
Office, TEMP coordination is normally carried out by the T&E Manager in the 
Project Office. CTE as the OT&E authority provides the OT&E input. The TEMP 
is endorsed by all affected organisations and is released jointly by the Deputy Chief 
of Naval Staff (DCNS), and the Assistant Chief of Naval Staff-Materiel (ACMAT- 
N), as the Developing Agency. If any issues remain unresolved at this level, the 
TEMP will be resolved by the Chief of Naval Staff (CNS). [Ref. 44:pp. 4-5] The 
authority for DCNS to release the TEMP was based on his role as Project Sponsor. 
With HQADF now assuming this role, the DCNS function in TEMP development 


is unclear. 
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In the USN, drafting the TEMP is the responsibility of the PM while the 
DoD heads of OT&E and DT&E authorities approve the TEMP. In contrast. in the 
RAN, the Sponsor has ultimate responsibility, although the TEMP is managed by 
the PM, and the TEMP is approved by the heads of the developing agency and the 


(former) Sponsor. 


b. TEMP Timing 


The TEMP is required in the at Milestone I- Concept Demonstration 
Approval within the USN, and is updated for each subsequent milestone and when 
significant program changes occur. The RAN requires the TEMP to be raised in 
draft form as early as practicable, but in actuality the TEMP for major projects is 


usually drafted after production approval (Milestone 3). 


6. Combined DT&E and OT&E 


“Combined testing” refers to a single test program conducted to support 
both DT&E and OT&E objectives. The advantages of combined testing is the 
shorter time required for testing, and cost savings by eliminating redundant 
activities. These need to be weighed against the limitations of the additional 
extensive coordination required and the less than optimum environment and 
coverage for OT&E that may occur. Early involvement of OT&E personnel during 
system development increases their familiarity with the system and permits 
identification of operational concerns early in the program. 

The USN has a policy that combined DT&E and OT&E testing should be 
considered when there are time and cost savings [Ref. 24:p. 8-3]. However, this 
combined approach must not compromise either the developmental or operational 
test objectives. Also a final independent phase of operational testing and evaluation 
is required for beyond low rate initial production decisions [Ref. 24:p. 8-4]. The US 


has extensive experience with combined testing, some successful and some 
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unsuccessful, and they conclude that it is possible to have combined test teams 
involved throughout the testing process. The DI&E and OT&E teams can share 
mutually beneficial data, as long as the test program is carefully planned. However, 
evaluation and reporting activities must be conducted separately. [Ref. 18:pp. 17-1 
- 17-4]. 

The conduct of combined DT&E, (PAT&E) and OT&E is not addressed in 
RAN T&E policy, however, it is implied as being almost a requirement. The T&E 
DI(N) [Ref. 44:p. 3] states that OT-3 and OT-4 will incorporate the requirements of 
PAT-6 and PAT-7 Inspection Tests and Trials as described in ABR 1921. It also states 
that OT-3 will be conducted during Ship Qualification Trials. The RANs “blurring” 
of the distinction between the types of T&E leads to DI&E and OT&E often being 
conducted over the same period, by the same test team. Although the RAN 
recognises, by definition, the difference in objectives and methodology between the 
types of T&E, no limitations or guidance as to the possible hazards of this combined 


testing approach are addressed. 


7. Modeling and Simulation 


The use of modeling and simulation can increase the efficiency of the T&E 
process, reduce the time and cost, provide otherwise unattainable and 
unmeasureable data, and provide more timely and valid results. The USN 
encourages the use of simulation and modeling to assist in projecting operational 
effectiveness and operational suitability prior to Milestone II - Development 
Approval, but limit the use of simulation and modeling in subsequent OT&E to that 
of supplementing OT&E test data [Ref. 15:p. 2-37]. 

The use of models and simulation in OT&E is not addressed by the RAN 
documentation, although modeling and simulation performed by the Maritime 


Systems Division of the Defence Science and Technology Organisation were used 
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by RANTAU to assist the evaluation of the upgraded combat and weapons systems 
in the Modernised DDGs. 

With the increasing costs of weapons systems, decreasing budgets and the 
subsequent need for efficiency in the conduct of OT&E, simulation can be used 
during OT&E to extend test results, to improve the statistical sample, or to 
determine overlooked or directly unmeasured parameters. This is particularly 
relevant to the RAN who lack appropriate ranges and the financial resources for 
extensive test programs of expensive weapons. A policy directing the use of models 


and simulation in OT&E in the RAN is required. 


8. Foreign Weapons Evaluation 


The US DoD has a foreign weapons evaluation program which is designed 
to support the evaluation of a foreign nation’s weapons system, equipment or 
technology in terms of its potential to meet a specific U.S. military requirement. The 
primary objective of the program is to reduce the costs of research and development, 
while leading to the acquisition of foreign equipment for U.S. use [Ref. 18:p. 21-1]. 
From the OT&E viewpoint, the USN ‘try-before-buy policy’ is still maintained, 
despite the early phases of OT being unable to be achieved. When procurement of a 
foreign weapon system is planned, the developing agency and COMOPTEVFOR 
may be directed to assess the adequacy of any previously conducted DT&E and 
OT&E and to provide recommendations on the need for additional T&E prior to 
procurement [Ref. 15:p. 2-34]. 

Despite the most recent Australian Government “White Paper” on Defence 
Policy [Ref. 6:p. 70] requiring the need to be able to determine the performance in 
the Australian environment of equipment of both overseas and local origin, no DoD 
or RAN T&E policy specifically addresses the evaluation of foreign systems. Prior 


to a major acquisition, project teams review possible contractors’ proposals and 


evaluate them against the user requirement. OT&E personnel may be part of these 
teams, however, there is no policy as to the OT&E authority’s role in foreign 


weapons evaluation within the RAN. 


9. OT&E Funding 


OT&E is an expensive process and generally occurs late in the life of a 
project when money is usually tight. Within the USN, funding associated with T&E 
(including instrumentation, targets and simulations) are identified in the system 
acquisition cost estimates, acquisition plans and the TEMP [Ref. 18:p. 18-11]. The 
Program Manager plans, programs, budgets and funds the costs of all resources 
identified in the approved TEMP for all T&E through OT-III. Funds required to 
conduct OT&E are programmed and budgeted by OPTEVFOR and advised to the 
Program Manager. OT&E costs include test articles, expendables, targets, data 
collection and reduction and OPTEVFOR program related costs. The Program 
Manager does not fund fleet operating costs for T&E support, which includes fuel 
and aircraft, These costs for OT-II and III, and all costs for OT-IV, except 
procurement costs and OPTEVFOR costs, are funded by the Fleet CINCs. [Ref. 
11:p. 21] The RAN employs a similar funding arrangement where OT&E is funded 
separately within a project and is estimated and bid for by CTE.[Ref. 44:p. 6] 

The essential difference is that the USN develops the TEMP. and hence 
identify the resources required and their costs, earlier in a project’s life than does the 


RAN, thus enabling the project to include these requirements in project funding. 


10. Land Based Test Sites 


A Land Based Test Site (LBTS) is a facility that duplicates, simulates or 
stimulates the employment of a system’s planned operational installation and 
utilisation. Used primarily for the purpose of conducting DT&E, it is sometimes 


used to test system integration and overall performance. The USN often use LBTS 
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for the conduct of initial OT&E to gain an estimation of potential operational 
effectiveness and suitability, primarily to support Milestone II[A (Low Rate Initial 
Production) decisions and not the more stringent Milestone IIIB (Full Rate 
Production). 

RAN guidance for OT&E using land based test sites includes a statement 
that system centres and simulators will be employed for early stages of OTE if 
available [Ref. 44:p. 4], and that OT-2 may be conducted at a land-based test site 
[Ref. 44:p. 3]. Although LBTS are being developed for both the ANZAC frigate and 
COLLINS submarine projects, OT-2 in support of production decisions will not be 
conducted. The use of these LBTS for later OT&E was not planned in the original 
project schedules, and so is subject to them being used on a non-interfering basis to 
the contractor. 

The USN has demonstrated that early OT&E on LBTS can give an 
estimation of potential operational effectiveness and suitability, and hence identify 
potential operational problems early and minimising operational risk. Their use for 
OT&E during the ANZAC frigate and COLLINS submarine projects would appear 


to offer similar advantages to the RAN. 


11. Ship OT&E 
Both the USN and RAN recognise that ship acquisition projects will be 


treated differently to others in so far as protracted construction time usually 
precludes T&E of a prototype before the decision to proceed to production. Because 
the development and construction period for a major ship in the USN, normally 
precludes completion of initial OT&E on the lead ship prior to the production 
decision for follow-on ships, successive phases of OT&E are accomplished as soon 
as practicable to reduce risk and minimise the need for modification to follow-on 


units. Ship acquisition programs, therefore, usually have OT&E between Milestones 
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II and III, which consist of individual weapon systems testing and system integration 
at land based test sites. 

With the construction of lower risk, proven overseas ships adapted to 
Australian requirements, the RAN assessed the degree of technical risk as low. 
OT&E between Milestones II and III in the RAN was, therefore, not considered 


necessary. 


F. SHIP ACCEPTANCE 


Both Navies have similar policies and procedures for acceptance of ships from 
contractors and then into service. However, only the USN states the purpose of ship 
acceptance which is “to ensure delivery to the Fleet of complete ships, free from 
both contractor and government responsible deficiencies” [Ref. 35:p. 1]. 

Each Navy uses similar terminology, but these have different meanings which 
can create confusion. A comparison of terms applicable to ship acceptance is given 


in Table 17. 


1. Ship Acceptance Policy 


Ship acceptance and Fleet Introduction within the USN has traditionally 
been based only on the successful completion of PAT&E and material inspections. 
However, with the treatment of ships as complete systems, as for the DDG-51, Fleet 
Introduction is now based also on OTKE results. 

The RAN has also placed emphasis on PAT&E leading to Acceptance into 
Naval Service (AINS), although OT-3 has been included in the trials program 
leading to AINS. However, these trials conducted as OT-3, are really PAT&E / 
DT&E with minimal true OT&E. 
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TABLE 17: COMPARISON OF ACCEPTANCE TERMINOLOGY 


Acceptance | The legal act of accepting Delivery | When the ship is contractually | the ship is contractually 
custody of a new construction dedinsemnerd from the 
ship by the Navy upon delivery contractor’s ownership to the 
by a private shipbuilder. Goverment. 


Delivery The actual assumption of } Preliminary | Determination that the ship has 
custody by the Navy incident || Acceptance | reached both material and 
to acceptance. The date of (PA) support status sufficient to 
delivery from a private safely and effectively proceed 
shipyard is also the date of with Navy trials. It is usually 
acceptance. concurrent with Delivery. 


Fleet Approval by SECNAV that a |} Acceptance | Acceptance of the ship by CNS 


Introduction | ship class meets the | into Naval | as a fully operational unit of 
operational effectiveness, Service the Fleet, based on operational 
operational suitability, safety (AINS) performance and _ support 
and material standards for levels obtained at that time. 
service use. 


Operational | Carried out only of first of 

Acceptance | class to supplement the AINS 

(OA) assessment using additional 

Operational and = support 

expenence with the Flee 

refine the vessel and Naval 
Requirements. 





2. Ship Acceptance Procedure 
The procedure for USN ship acceptance and introduction to the Fleet as 
used for the DDG-51 program is outlined in Figure 23. A noteworthy aspect of the 
USN schedule is the exclusive OT-3 period, the results of which support a Fleet 


Introduction decision. 
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Figure 23: USN Ship Acceptance Schedule and Milestones 


The Board of Inspection and Survey (INSURV) conducts Acceptance Trials 
and Final Contract Trials for each individual new ship to provide independent 
verification of readiness of ships for acceptance in the USN. Based upon its findings. 
the Board recommends acceptance or final settlement of the contract. The Board 
identifies material conditions which represent departures from the USN General 
Specifications and deficiencies that substantially reduce the ship’s fitness for naval 
service or degrade its ability to perform its primary mission.[Ref. 34:p. 1] On 
successful trials completion, NAVSEA accepts the ship from the contractor. 

For comparison, the RAN ship acceptance schedule, as used for the 


Australian Frigates and proposed for the ANZAC frigates, is outlined in Figure 24. 
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Figure 24: RAN Ship Acceptance Schedule and Milestones 


Of note in this schedule is the combined PAT&E / OT&E period supporting 
an AINS decision, and the OT-4 phase leading to OA. OT-4 is not an exclusive T&E 
phase, but is integrated with the normal fleet operations of the ship. AINS is the most 
significant post delivery milestone as it represents acceptance of the ship into the 
RAN and that the project has met it’s responsibilities. 

In the RAN, Delivery (contractual acceptance) of the ship to the 
Government is a contractual matter and is managed by the Project Director. CTE’s 
responsibility is to conduct the Acceptance Trials and report the results. An 
Acceptance Board (AB) is established to provide an assessment of the ship and 
submit recommendations upon which CNS can base his acceptance decision, 
however the AB is quite different from the INSURV Board. The AB is usually 
established for each new ship type and only for first of class. The President, Vice- 


President and Board members serve on the Board part time. The AB’s sole function 


156 


is to advise CNS on acceptance matters, This is achieved by assessing the adequacy 
of the trials plans, examining the inspection and trials reports, attending trials where 
necessary and then providing an overall assessment of the adequacy of the ship 
against the user requirement. The Acceptance Board does not conduct tests, trials or 
inspections, and is essentially a board of review. 

Acceptance into Naval Service is essentially the transfer/acceptance of the 
ship from the acquisition part of the Navy (ACMAT-N) to the inservice operations 
and support areas, as having met the user requirement. Post delivery trials leading to 
AINS includes the SQT period, with harbour and sea phases. 

For the first of class, it will normally not be possible to trial and evaluate all 
aspects of the operational performance and maintainability prior to AINS. This OT- 
4 period is conducted following AINS and is compiled by RANTAU in consultation 
with the Naval and Maritime Headquarters Staff. It comprises a series of trials and 
evaluations to assess operational effectiveness and operational suitability. [Ref. 
4l:p. 15.14] Typically, the Follow-on T&E program will last from 12-24 months, 
depending on the class of ship. At the end of this period RANTAU compiles a report 
in consultation with Maritime Headquarters Staff. The AB assesses the results and 
provides the overview to ensure the program is comprehensive. This assessment 


serves to confirm or modify the assessment at AINS. 


3. Ship Acceptance Summary 


Both Navies now include OT&E in their post-delivery trials of the first of 
class to ensure that the ship is assessed operationally against the User Requirement. 
However the RAN OT-3 period is not true OT&E, but consists mostly of DT&E and 
PAT&E with some OT&E elements. The USN INSURV Board conducts acceptance 
trials on every ship but looks at materiel items only, which is different to the RAN 


AB, which is a board of specialists who review the trials and results of the first of 
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class. OT-3 within the RAN needs to be separated from the DT&E /PAT&E post 
delivery events, and should consist of true OT&E. This would enable the 
Acceptance Board to use its knowledge and experience in assessing the results of 
OT&E against the user requirement, instead of assuming a quality control function 


over what is essentially PAT&E testing. 


G. OT&E PROCEDURES 


This section discusses the procedures adopted by the two Navies in 


implementing OT&E policy. 


1. User Requirement to OT&E Test Plan 


To this concruence between OT&E test documentation and the user 
requirement, both Navies generate a series of documents through the acquisition 
process. Although the documents between Navies have different titles, they have 
similar functions. A comparison of the progression of documentation leading from 
the user requirement to the OT&E test plans is given in Table 18. 

Both Navies can trace a requirement from the capability need through the 
refining requirements documentation to specific a objective in the TEMP, to the test 


plan and ultimately to the final OT&E report 
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TABLE 18: COMPARISON OF REQUIREMENTS DOCUMENTATON 


Document 

aan Function en 
Mission | Mission Need Statement Statement | Capability | DefenceForre | Force 
(MNS) Need Capability Proposal (DFCP) 
Operational Requirements Refined Required Operational 
Document (ORD) Requirement Characteristics (ROC) 
Top Level Requirement Requirement Agreed Ships 
GnIER) Specifics Characteristics (ASC) 
Test and Evaluation Overall Test and Evaluation 
Master Plan (TEMP) T&E Plan Master Plan (TEMP) 
OT&E OT&E Acceptance Management Plan 
Test Plan Plan (AMP) 


2. Assessment of Operational Effectiveness 









The RAN has found it difficult to define exactly what is required to assess 
operational effectiveness. The RANTAU guidance on the conduct of OT&E [Ref. 
53] recognises that the user requirement is the bench mark for determining the 
degree to which a product is effective. Also that operational effectiveness is best 
assessed by a performance demonstration by normal operating personnel in the 
normal or given environment. But otherwise provides very little guidance on the 
assessment of operational effectiveness. Past techniques employed involve 
analyzing each COI and employing modified Ship Qualification Trial and Fleet 
Exercise Program techniques to evaluate them. Use has also been made of 
simulation, weapon analysis and exercise analysis. The lack of suitable range 


facilities has also limited the assessment of operational effectiveness. 
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The USN, by comparison, has a more effective method of assessing 
operational effectiveness. As do the RAN, the USN examines each COI and related 
operational effectiveness objective, and decides what needs to be known to enable 
each objective to be assessed [Ref. 15:p. 9-1]. They ensure that the appropriate 
environments, threats, etc. are included and that sufficient data will be generated to 
address the COI and objectives. They focus on achieving statistical relevance where 
possible, and making the tests as objective as possible. DOT&E works closely with 
the USN to help determine how many test assets are required to achieve a certain 
degree of confidence that the results are correct. A test may need to be repeated a 
number of times to gain statistical relevance. However, if this is not possible, or the 
direct proving of an ability is not possible (as in the surface warfare example), then 
the test needs to be designed so that what is observed in the test program is projected 
to be what would be observed in reality. (Ref. 19][Ref. 21] These situations require 
expertise and judgement, and compromise between the authorities involved, to 


make the tests as objective as possible. 


3. Assessment of Operational Suitability 


The principles of operational suitability are common between the USN and 
RAN, and similar aspects are assessed. Within the USN, COMOPTEVFOR reviews 
the Integrated Logistic Support Plan (ILSP) against the user requirement. Following 
this review, COMOPTEVFOR determines the suitability tests required based on 
expected reliability, degree of confidence, thresholds, etc. required to be achieved. 
These determine the scope and length of the assessment. Reliability, maintainability 
and availability (RM&A) data is collected and sent to the normal inservice analyst 
of RM&A data, the Naval Weapons Analysis Centre (NWAC) for analysis. 
Specialist analysts within OPTEVFOR design the tests and determine measures of 


suitability. They also evaluate and analyse the adequacy of logistic supportability. 
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However, OPTEVFOR also relies on operational personnel to use their experience 
and knowledge of the system to identify inadequate logistic support. 

The assessment of operational suitability in the RAN has been 
controversial, particularly the assessment of supportability. Before 1987, the 
assessment of operational suitability, and ILS in particular, was rather subjective and 
often controversial. A new approach was taken in 1987 and has continued to be 


refined. The assessment of ILS is now conducted in two phases: 


e« ILS Structural Review 
¢ ILS Operational Review 


The ILS Plan is reviewed and discussions with the functional areas 
handling the support.conducted prior to operational testing. During the operational 
testing period, the ship’s ILS related documentation and records are reviewed, 
maintainability demonstrations are performed and a supportability assessment is 
conducted. The RAN is currently developing a routine in-service RM&A data 
collection and analysis system, however, until that is operational, the data is 
collected and analysed on an ‘as required’ basis by the Trials Authority. 

In summary, the principles of operational suitability assessment are 
common between the USN and RAN. Although assessments by the RAN have 
become less objective and more subjective over recent years, the assessment of 
RM&A is still in its infancy. Both Navies recognise the importance of data 
collection, however, the RAN needs experienced personnel to design tests and 


analyse results. 


4. OT&E Coordination 


The USN recognises the importance of coordination between authorities in 


the successful achievement of T&E. The USN T&E coordinator is responsible to the 
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Director, Test, Evaluation and Technology Requirements (NO91). They are 
essentially the OPNAV Staff coordinator for Navy T&E,. providing a primary 
contact point for all parties and setting up T&E briefings and meetings. Their 
responsibilities include the chairmanship of the Test and Evaluation Coordination 
Group (TECG) for each major program.{Ref. 37] TECGs are used for complex, 
multifaceted programs which require extensive T&E coordination. Membership of 
a TECG includes the Program Manager, the Sponsor, COMOPTEVFOR, a logistics 
coordinator and others as appropriate (such as a PREINSURV representative). 
TECG recommendations are considered for inclusion in the TEMP. [Ref. 11:p. 15] 
Some of the functions of a TECG are the early definition of terms, measures of 
effectiveness and how these are to be measured, and the criteria for acceptable or not 
acceptable. Of note is the formation of a TECG does not imply a joint test team 
approach. Each T&E agency remains fully and solely responsible for conducting 
and reporting the types and phases of T&E for which it is accountable. 

In the RAN, a T&E Planning Group (TEPG) is formed prior to TEMP 
development under the leadership of the T&E manager for the project. Foundation 
members of the TEPG are the Project Director, Project Sponsor, the Design 
Approval Authority and RANTAU, with other parties seconded/called upon as 
required. [Ref. 44:p. 5] The TEPG analyses T&E requirements and estimates 
resources required to meet the requirements. 

The function of these groups is similar, however, the USN TECG is led by 
the Director, Test, Evaluation and Technology Requirements (NO91) under the 
CNO, whereas the RAN TEPG is led by each individual Project Director through 
this T&E manager. Despite their best intentions, Project Directors are essentially 
driven by cost and schedule considerations, not the overall T&E adequacy of their 
project. To be truly objective, each TEPG should be chaired by the authority 


responsible for ensuring the adequacy of the RAN’s overall T&E program. 
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5. OT&E Reporting 


Reports provide the OT&E authority’s conclusions regarding a system’s 
operational effectiveness and suitability, and his recommendations regarding the 
systems future, i.e., acceptance, further development, additional OT&E, etc. The 


USN system has three basic principles for OT&E reports: 


e Impartial 
¢« Complete and Thorough 
¢ Sole Reporter 


The conduct of OT&E must be impartial and not influenced by the program 
manager or contractors during testing and analysis. COMOPTEVFOR’s 
conclusions are based on a complete and thorough analysis. A report covers a 
complete OT&E phase (e.g., OT-IIIA), relating the test results to the COIs and 
addressing the objectives stated in the TEMP. Quick-look and interim reports are 
usually sent only if the testing could not be completed or when directed by the CNO. 
Although the final report may be a surprise to other authorities. OT&E results and 
analysis are reported through one authority only. Comments from the Commanding 
Officer of the ship under test ship, for example, are sent only to COMOPTEVFOR. 
All operational test data is considered to be the owned by COMOPTEVFOR until 
the final report is signed. 

Within the RAN, the OT&E Authority coordinates the issue of trials 
reports, however, they are issued on completion of each segment of OT&E e.g., o/c 
ASW phase of OT-3 etc. Quicklook reports are also routinely sent on completion of 
each week’s testing. The ship under test also often provides a report, however, it is 
usually forwarded to their operational authority and the Project in addition to the 
Trials Authority. For the results of PAT&E, rapid feedback is necessary to ensure 


timely rectification of defects. However, OT&E results may indicate system 
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deficiencies against user requirements rather than equipment defects, so the results 
need to be fully analysed and the implications assessed before being reported to a 
wider audience. The RAN practice of OT&E reporting probably stems again from 
the lack of delineation between types of T&E and the combined PAT&E / OT&E 
nature of the RAN’s OT-3 testing. 


6. OT&E Personnel Selection and Training 


OT&E is a specialised discipline, with its own philosophy and 
methodology, and so requires a specialist approach with knowledge and experience 


to make it effective. 


a. Selection 


The U.S. community involved with ship OT&E select and conduct 
limited training for their military personnel. Although there are no special selection 
requirements for detailing military personnel for service as Test Directors in 
OPTEVFOR, broad ship operations experience and combat system knowledge are 
preferred. For particular programs, OPTEVFOR make use of subject area experts, 
either resident or borrowed from a non-interested party. Military personnel required 
by DOT&E are selected by background and expertise, preferably war college 
graduates with joint experience. 

No special selection criteria is involved for posting OT&E personnel 
within the RAN, although like the USN, broad ship operations experience and 
combat system knowledge are preferred. Very few personnel have had any T&E 
experience prior to joining the OT&E authority. Also currently the OT&E authority 


is viewed by some as not being career enhancing for warfare personnel. 
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b. Training 

COMOPTEVFOR conducts a four day Operational Test Director’s 
(OTD) overview course covering the major areas of OT&E. Further details of this 
course are included in Appendix A. They also run adhoc segment courses which 
provide acquaints or updates on OT&E subjects e.g., analysis, test plan development 
and threat updates. To assist their personnel in managing OT&E, OPTEVFOR 
publishes the OT&E Director’s Guide which documents their philosophy and 
methodology. (Ref. 21] Other authorities involved with OT&E also send personnel 
to attend this course. Although DOT&E does not conduct courses for new staff, a 
one month handover is usually required. 

The RAN has no training courses on OT&E, or on T&E in general. A 
number of officers have completed the USN Operational Test Directors Course in 
recent years which has improved the knowledge and understanding of OT&E within 
CTE’s organisation considerably. These courses have been arranged on an ad-hoc 
basis through overseas visit submissions rather than as pre-requisite courses for 
particular billets. For guidance of trials personnel and other authorities, CTE has 
developed a number of Standard Operating Procedures addressing certain aspects of 
Oltec E., 


c. Career Path 


Although some personnel may develop into OT&E specialists, no 
career path in OT&E is consciously provided in the USN, and no naval personnel 
currently serving in DOT&E have served previously in OPTEVFOR. Similarly in 
the RAN, no career path in T&E is provided. 

To be effective, OT&E ideally requires current, experienced, 
knowledgable personnel with high professional credibility within their field of 


expertise. To achieve this, selection of suitable personnel, adequate training and 


165 


good guidance documentation is required, and preferably a career path where OT&E 


is seen as suitable career progression by such capable personnel. 


H. OT&E IMPLEMENTATION 


Recent ship projects illustrate how these policies and procedures are 
implemented. The USN DDG-51 program employs a new hull with a new, 
evolutionary combat system. Noteworthy aspects of DDG-51 OTE included OT- 
II on both combat and propulsion systems LBTS to gain an early indication of 
potential operational effectiveness and suitability which supported the decision to 
approve the DDG-51 to go beyond Low Rate Initial Production. Another 
characteristic of the DDG-51 program was “Whole Ship” OT&E where the whole 
ship is viewed as an integrated warfare system, rather than as a platform for 
individual warfare systems. OT-3 was conducted in a dedicated period following 
CSSQT and Final Contract Trials (but before the workup) to support a Fleet 
Introduction decision. It consisted of fully assessed, free play multi-threat scenarios 
with the emphasis on operational realism. 

The implementation of OT&E policy within the RAN is best demonstrated by 
reviewing examples of ship OT&E conducted to date, and that planned for ships due 
to enter service in the near future. 

The three RAN DDGs underwent a major modernisation and refit over the 
period 1987-1991. The major modernisation occurred with the ships’ combat 
system, command and control system, sensors and weapons, with the aim of 
increasing the ‘supportability’ of the systems rather than increasing the ships’ 
capability. As a low operational risk project, the OT&E followed the established 
RAN policy, however, it was the first to actually implement an OT-4 period to more 


fully assess operational effectiveness and operational suitability leading to OA. 
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Two US FFG-7 class frigates, known as Australian Frigates (AF), are being 
built in Australia. The AF is a low operational risk project, except for the addition 
of the Mulloka sonar. The OT&E followed the established RAN policy, however, it 
failed to plan an OT-4 period to more fully assess operational effectiveness and 
operational suitability of the Mulloka sonar. 

Australia and New Zealand are cooperating in the building of the ANZAC 
frigate. Essentially a MEKO 200 design incorporating a combination of European 
and US equipment, the design is classified as low risk, although modifications to the 
design were necessary to meet RAN requirements. Although only in the early 
stages, the planned OT&E for the ANZAC ship follows the established RAN policy, 
with the OT-3 period combined with the SQT, being conducted prior to AINS. The 
ship class has a higher operational risk than either the modernised DDG or the AF. 

The RAN is building six boats, known as the Collins class, to the Swedish 
Kockums Type 471 design. They will be the first submarines to be constructed in 
Australia and will have a unique equipment fit These submarines possibly have the 
highest operational risk of all current major RAN combatant projects. Consequently 
the increased need for OT&E has been recognised. The planned OT&E program 
demonstrates a departure from the usual RAN OT&E policy, and reflects a growing 
awareness of the need to separate DT and OT activities, and to include a dedicated 
OT-3 period after the usual SQT and ORE. 

These examples of RAN implementation of ship OT&E policy illustrates a 
number of key issues. The operational risk of ship projects within the RAN is 
increasing, leading to a growing awareness for the increased need for ship OT&E. 

Significant differences in OT&E implementation between the USN and RAN 
are, first, that the RAN does not generally conduct OT&E to support the production 


or purchase decision of ships. Second, the RAN has relied on the results of a 
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combined SQT and OT-3 period to provide a basis for AINS, whereas the USN 


conduct a “whole ship” dedicated OT-3 phase to support Fleet Introduction. 


I SUMMARY 


The USN and RAN have similar acquisition systems and similar fundamental 
definitions of T&E. However, the OT&E systems differ in a number of significant 
areas. Both Navies appreciate the importance of T&E in the acquisition process, 
however, the level of importance of T&E, and OT&E in particular, differs greatly. 
The USN recognises that without OT&E, it risks deploying systems which cannot 
adequately perform significant portions of their missions, thus degrading its 
deterrent/defensive capabilities and endangering the safety of military personnel 
who operate and maintain the systems. The importance of OT&E within the RAN is 
not specifically addressed in any T&E policy document. The USN places emphasis 
on OT&E leading to full scale production where the majority of a project’s funding 
is spent. In contrast, the RAN system places a high priority on the post production 
phase. 

The RAN has adapted the more detailed USN T&E definitions to suit its own 
requirements. During this process, the RAN has lost the distinction between the 
types of T&E and often confuses OT&E with PAT&E / DT&E. USN OT&E policy 
has its basis in statutes approved by Congress. The RAN has no such formal basis 
and its OT&E policy is embedded in the more general T&E policy. 

The USN OT&E organisation is well structured with comprehensive 
responsibilities for each participant, and clearly delineated from the DT&E 
structure. Following a number of recent reorganisations, the RAN’s OT&E structure 
is in disarray, with no clear policy maker for T&E in general, or OT&E in particular, 
lacks functional authorities with T&E knowledge to advise Project Directors, and its 


OT&E authority is buried in the in-service area of the Maritime Command. 
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Both Navies have similar policies and procedures for acceptance of ships from 
contractors and then into service. Each Navy uses similar terminology, but these 
have different meanings. OT&E results in the USN is now having a greater input on 
ship acceptance. Although the RAN has traditionally included OT&E in support of 
AINS, the nature of the testing has really been more DT&E/ PAT&E rather than true 
OT&E. 

The USN OT&E system has, by necessity, developed into a well organised, 
well documented and effective system. The RAN system, on the other hand, suffers 
from confusing T&E policy documentation, a weak OT&E organisation structure 
and a general lack of OT&E knowledge and appreciation. The determination of the 
OT&E required has traditionally not been achieved early enough in the life of a 
project, and so project funding has not included the provision for OT&E. However, 
OT&E awareness and knowledge within the RAN is growing as evidenced by the 
proposed Collins class post delivery trials. 

The RAN OT&E system, although originally based on USN OT&E philosophy. 
now differs in the importance, interpretation, application and focus of OT&E Asa 
result of organisation changes, it now lacks the ability to effectively manage overall 
T&E, let alone OT&E, within the RAN. To achieve efficient and effective trials and 
acceptance of the new higher risk combatants currently under construction, the RAN 


OT&E system needs to be revised to fully support the acquisition process. 
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VI. AMODEL FOR OT&E IN THE RAN 


This chapter proposes a model for the OT&E system in the RAN. The OT&E 
principles derived from the USN system are used to develop a system to suit RAN 


requirements. 


A. TEST & EVALUATION DEFINITIONS 


The basis for the OT&E system are clear definitions of the types and phases of 
T&E. The RAN should continue to recognise the three types of T&E associated with 
ships. The current definitions listed in the CPO Manual [Ref. 38] should be amended 


to better suit the Australian Defence environment. 


1. Development Test and Evaluation (DT&E) 


The DT&E definition remains unchanged as follows: 


DT&E is conducted to assist the engineering design and development process, 
and to verify attainment of technical performance specifications and 
objectives. 


The proposed phases of DT &E are detailed in Table 19. The three phases 


TABLE 19: PROPOSED PHASES OF DT&E 


Current RAN Description | Current RAN Description | DT | Proposed RAN Description Proposed RAN Description 
Validation of design concept. [vats of cesienconeers_|/DT | Valent Validation of design concept. 
Proving design. Demonstration that design meets specifications. 


Demonstration that production | DT-3 | Demonstration that production meets required 
meets required technical technical characteristics or establish standards for 
characteristics. first of class. PAT&E is a form of DT-3. 
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are retained, however, the DT-2 definition is amended to clarify the meaning of 
“proving design’, and the DT-3 phase now recognises that PAT&E is a form of 
DT&E. 


2. Production Acceptance Test and Evaluation (PAT&E) 


The definition of PAT&E is revised to recognise that it is conducted not 
only during the contract period, but also during subsequent RAN testing: 


PAT&E is conducted on production items to ensure systems meet technical 
specifications and requirements, and is a type of DT&E. 


The proposed phases of PAT&E are listed in Table 20. Phases PAT-0 to PAT- 
5 remain unchanged. PAT-6 is amended to delete any connection with OT-3, as under 
the proposed model OT-3 will now be conducted separately. The PAT-7 phase is 
deleted as no PAT&E is conducted during the OT-4 phase of testing. 


TABLE 20: PROPOSED PHASES OF PAT&E 


development tests development tests 


System development tests PAT-3 System development tests 


| Seatesting == testin | Seatesting == | PATS | | Seatesting testin | Seatesting 


Operational and Qualification Tnals = Certification and Qualification 
(conducted with OT-3) Tnials 

Follow on Operational T&E PAT-7 : 

(conducted with OT) 


ha 








3. Operational Test and Evaluation (OT&E) 


The definition of OT&E is revised along USN lines, so that it more 
accurately describes the purpose and nature of OT&E: 


OT&E is conducted to determine a system's operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 
modifications to meet established OT thresholds, and develop tactics. OT&E 
has three distinguishing characteristics: 


e It is conducted in an operationally representative environment. 


¢ [tis conducted on production representative equipment using fleet personnel 
for operation and maintenance. 


¢ It is conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 


The proposed phases of OT&E are described in Table 21. These amended 
definitions include reference to the acquisition milestones they support. The 
reference to OPEVAL in the OT-2 phase has been deleted, as the RAN uses the term 
OPEVAL as an abbreviation for an operational evaluation in any OT phase. The 
reference to a prototype in OT-2 has also been deleted as OT-2 may also be 
conducted in a LBTS, or on a sample system if the proposed purchase is “off the 
shelf’. The OT-3 phase is revised to delete the Certification /Qualification Trials as 
they are now recognised as PAT&E. The definition of OT&E includes the use of 
normal fleet personnel so this wording has been deleted from the phase definitions. 
The OT-5 phase is amended to cover all in-service OT&E which could be conducted 
to assess new applications, new tactics, revised threat, etc. Reference to defect 
rectification is deleted also as it is a sub-phase of OT-4, and the assessments of 
modernisations and modifications are deleted as they would be new OT-2 or OT-3 


phases. 
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TABLE 21: PROPOSED PHASES OF OT&E 


Current RAN Description Proposed RAN Description 


Operational assessment of the Operational assessment of the 
development proposal development proposal to support Full 
Scale Engineering Development 
approval. 



































Demonstration of achievement of 
program requirements for operational 
effectiveness and suitability to support 
a production or purchase decision. 


Demonstration of achievement of 
program requirements for operational 
effectiveness and suitability of a 
prototype (OPEVAL) to support 
proceeding to full production 























Demonstration of achievement of 
program requirements for operational 
effectiveness and operational 
suitability on production of ship/ 
aircraft/system to support Acceptance 
into Naval Service. Includes limited 
reliability, maintainability, availability 
and logistic supportability 
assessments. 


Demonstration of achievement of 
program requirements for operational 
effectiveness and suitability on 
production of ship/aircraft/system, 
normally in independent operations 
using normal Fleet personnel. 
Certification/Qualification Trials 
moelude “limited reliability, 
maintainability, availability and 
logistic supportability assessments. 


































Demonstration of achievement of 
program requirements for operational 
effectiveness and operational 
Suitability on production of ship/ 
aircraft/system ina multi-force, multi- 
threat environment to support 
Operational Acceptance. Includes 
detailed reliability, maintainability, 
availability and logistic supportability 
assessments. 


Demonstration of achievement of 
program requirements for operational 
effectiveness and operational 
suitability on production of ship/ 
aircraft/system using normal Fleet 
personnel in a multi-force, multi-threat 
environment, Includes detailed 
reliability, maintainability, availability 
and logistic supportability 
assessments. 
































In-service OT&E, which could include 
new applications, new tactics, revised 
threat, etc. 


Follow-on OT for assessment after 
modernisations, new applications or 
defect rectifications after OT-4. 
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4. T&E Types in Total Ship Test Program 


The Total Ship Test Program (TSTP) concept is now well established in 
RAN shipbuilding projects, although modified from the U.S. system. These refined 
T&E phases can be incorporated into an amended TSTP. However, instead of being 
grouped together as in the current TSTP, the DI&E and OT&E events are separate 
categories of testing. Because these separate DT&E, PAT&E and OTE events are 
integrated within the TSTP, these trials are termed collectively as “Integrated Tests 
and Trials’, which is an extension of the current RAN application of this term. The 


proposed categories of ITT are listed in Table 22. 


TABLE 22: CATEGORIES OF ITT 


A G 
[Fares [ rat [vicanent gteion 
seats [Siem aescomenress 
Se 0 
as [ats Ying 
fara [ore [omni 


Note: * Only performed on first of class 










The Certification and Qualification trials (ITT-6) are classified as DT &E for 
the first of class only, as standards of performance are established. For follow-on 


ships these trials are classed as PAT&E, as the standards established by the first of 


174 


class become the benchmark to be achieved by follow-on ships. ITT-0, ITT-2, ITT- 
3 and ITT-7 tests are performed only on the first of class. ITT-8 testing would only 
be performed once for a ship class, but may be conducted progressively on any ship 


of the class. 


B. T&E IN THE ACQUISITION PROCESS 


To ensure the progressive assessment of operational effectiveness and 
operational suitability during the acquisition process, OT&E is scheduled towards 
the end of each acquisition phase, usually after the DT phase. The proposed T&E 
schedule is shown in Figure 25. No change is suggested to the current relationship 
between OT&E phases and acquisition milestones, however, the figure separates 


DT&E/PAT&E from OT&E to clearly differentiate these different types of testing. 


FULLSCALE 
PROGRAM ENGINEERING PRODUCTION PRELIM aws 
INITIATION DEVELOPMENT APPROVAL ACCEPT 
APPROVAL 


PHASE 1 PHASE 2 PHASE 3 PHASE 4 PHASE 5 


DEMONSTRATION| FULL SCALE 
AND ENGINEERING PRODUCTION IN-SERVIC E 
EV ALUATION DEVELOP MENT 
PAT&E | DT-3/ 
1 T-2 ATT 0-5) PAT&E 
~~ ae 
OT&E 
OT-3 OT-4 OTS 
ATT 7) (TT 8) 





Figure 25: Proposed Test and Evaluation in the Acquisition Process 


ss 


As the construction of proven overseas ship designs adapted to RAN 
requirements, does not require the conduct of OT-2 prior to a production or purchase 
decision, the first opportunity to evaluate potential operational effectiveness and 
operational suitability may be in a LBTS after the production decision has been 
made. To ensure consistency of terminology, this testing should not be termed OT- 
2 as it does not support a production or purchase decision. It should be classified as 


the first phase of OT-3. 


C. RAN OT&E ORGANISATION 
The objective of an OT&E organisation structure is to provide effective OT&E 
to support the acquisition process, and hence the combat effectiveness of the RAN. 


The proposed OT&E structure is shown in Figure 26. The effectiveness of OT&E 


MINISTER 
FOR 
DE FENCE 
SECDEF 
DG FDS EA) 


/ 
| 
| | MARITIME 
! OMMANDER 
PROJECT 
COMMANDER INSERVICE 
OT&E 


Figure 26: Proposed RAN OT&E Organisation 
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can be achieved by devoting appropriate resources of both personnel and funding, 
by considering the requirements of all functional areas concerned with OT&E, and 


by coordinating these different functions. 


1. Policy Maker and Overseer 


An authority is required to formulate OT&E policy, and to evaluate and 
oversee the OT&E process. OT&E policy should come from an operational, not 
technical, functional area. HQADF now contains the centralised force development 
and user requirements organisation. As OT&E is the feedback of the performance of 
the final product against the user requirement, then ideally, the head policy maker 
and overseer of OT&E should be within HQADF or DoD, along the lines of the U.S. 
system. 

A shorter term, RAN-only, solution would be to include the overseer and 
policy maker function within the revised Director General Naval Policy and Warfare 
(DGNPW) or ganisation under DCNS. Although this area has recently been reduced, 
DGNPW still retains the responsibility for naval warfare information and expertise. 
The Director of Naval Warfare (DNW), under DGNPW, provides an information 
brokerage service to senior level management. He currently has a secondary 
function of providing naval warfare input to the Defence Capability Proposals 
developed by HQADF, and providing advice on naval warfare operations, 
equipment and personnel aspects. [Ref. 55:p. A-4] The responsibility of providing 
advice and input on OT&E matters, as part of the OT&E policy maker and overseer 


function for the RAN, is seen as a natural extension to these current responsibilities. 


2. Sponsor 


The Sponsor of major naval acquisitions is DGFD(Sea) within HQADF. 
Traditionally in the RAN, it has been the Sponsor’s responsibility to initially 


determine the T&E requirements in an acquisition program and to detail those 
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requirements in the capability proposal. To be able to determine the appropriate 
nature and extent of OT&E required, the Sponsor needs access to OT&E knowledge 
and experience. A close relationship needs to be maintained by DGFD(Sea) with the 
OT&E policy maker and the OT&E authority (Note dotted line). 


3. Developing Agency 

The Materiel Division under ACMAT-N, performs the Developing Agency 
function within the RAN. Currently major combatant projects establish a TRE 
manager, with the larger ones also establishing a manager exclusively for post 
delivery trials. For example, the Australian Frigate Project has a Post Delivery Test 
and Trials manager, and the New Submarine Project has just established a 
Submarine Project Transition and AINS manager. However, ACMAT-N does not 
have a central T&E policy authority to advise these T&E managers. Although the 
Director of Naval Equipment Production (DNEP) has responsibility for providing 
specialist technical advice on technical/engineering activities, which includes trials 
and acceptance of equipment [Ref. 64], he specialises more in advice for contract 
management than for post delivery T&E. 

To assist all RAN projects with T&E matters, an authority is required 
within the Materiel Division whose responsibilities would include assisting Project 
Directors in complying with policies, incorporating lessons learned from previous 
projects and reviewing the T&E aspects of projects at major milestones. Although 
primarily a DT&E policy authority, this office would also have an input into the 
OT&E process by reviewing the TEMP, and by advising projects how to prepare for 
OT&E. The Director General Naval Engineering Requirements (DGNER) is 
currently responsible for the design, development, acceptance into service and 
through life support of ships. This engineering policy directorate appears to be well 


suited to the DT&E policy and review function. 
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4. OT&E Tester & Evaluator 
Ideally, the OT&E Tester & Evaluator should be independent and focus 


exclusively on OT&E. He should also have operational credibility and influence in 
operational matters relating to OT&E. The current OT&E Tester and Evaluator, 
Commander, Test and Evaluation (CTE), within the Maritime Command is 
independent from developing and production agencies, but is responsible to the end 
user of the system. This arrangement has advantages to CTE of access to 
knowledgeable and experienced fleet staff warfare officers and fleet planners for 
assistance in trials planning and assessment. However, there are possible 
disadvantages to this lack of independence from the end user. The Maritime 
Commander may have aims and objectives which could conflict with the total 
impartiality of OT&E reporting. As the Maritime Commander establishes 
scheduling priorities for his resources, priority given to OT&E is subject to the 
politics of requirements and funding. OT&E may take a low priority behind 
operations and training. The Maritime Commander may also not wish to “‘stress” the 
system during OT&E, in case he perceives that operational deficiencies so 
highlighted may be seen as having some reflection on his own organisation. 

CTE is currently multifunctional, conducting DT&E and PAT&E in 
addition to OT&E. This results in a lack of differentiation within CTE’s organisation 
between DT &E/PAT&E and OT&E. Also being placed under the inservice 
engineering area of the Maritime Command, he lacks visibility and influence within 
the warfare community and hence credibility in operational matters critical to 
OT&E. CTE’s OT&E responsibilities are overshadowed by his DI &E/PAT&E 
duties. 

To achieve the advantages of DT&E and OT&E separation, a revised 


organisation is proposed, similar to that outlined by the former RANTAU in 
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Reference 63. All CTE’s current DT&E/PAT&E responsibilities, and those already 
the responsibility of CSO(E), would remain with an in-service trials team under 
CSQ(E). This trials team, headed by an engineering branch officer, would be formed 
from the current CTE technical personnel, and could also conduct certifications, 
Light Off Examinations, and the technical aspects of Ship Qualification Trials. The 
team would provide technical assistance and advice to a separate OTE trials team. 

CTE’s OT&E duties would be removed and managed by a separate group, 
Commander, OT&E (COT&E), working under an operational AY titty There are 
two such authorities within the Maritime Command; the Chief Staff Officer 
(Operations) (CSO(O)) who is responsible for plans and operations, and 
Commodore Flotillas (COMFLOT) who is responsible for Fleet readiness. As 
OT&E is the assessment of operational effectiveness and suitability, COT&E should 
be responsible to COMFLOT. Headed by a Warfare branch officer, COT&E would 
manage the post delivery trials period for all ships from the MHQ perspective. He 
would still manage the SQT, which, although not true OT&E, still has some 
operational input which could not be met by the in-service trials team under 
CSO(E). He would be the OT&E authority with responsibilities to manage all RAN 


OT&E including OT-3,which is now only conducted on first of class. 


5. OT&E Organisation Summary 
This proposed reorganisation achieves the aim of DT&E/OT&E separation 


with some concessions necessary due to manpower constraints. Also, by including 
OT&E under COMFLOT and managed by a Warfare Officer, OT&E could achieve 


the visibility, credibility and influence necessary within the RAN. 
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D. RAN OT&E POLICY IMPLEMENTATION 


1. OT&E Policy Documentation 

RAN T&E documentation needs to be clear, concise and comprehensive so 
that is serves as a valid reference and guide for all personnel involved with T&E. To 
correct the current inconsistent and out of date documentation, a major rewrite, 
rather than just updating is required. This rewrite of T&E documentation could be 
the vehicle by which the T&E system within the RAN is overhauled. Before the 
documentation is revised, T&E definitions and terminology need to agreed within 
the RAN and DoD. Then the OT&E and DT&E organisations within the RAN need 
to be resolved, and the appropriate policies decided. This could be achieved by the 
development of a revised RAN T&E DI(N), followed by ABR 1921 which would be 
firmly based on the T&E DI(N). 


2. Independence of OT&E Agency 
Given that the proposed COT&E is organisationally independent from the 


developing and production agencies, his procedural independence should also be 
achieved to ensure impartiality and validity of testing. When OT&E is conducted. it 
should not be influenced by the Project or DAA. No Project or DAA personnel 
should be present during testing. A final report only should be issued by the 
Maritime Commander, with no progress or interim report (unless the testing is 
unusually lengthy), to allow full analysis and evaluation of the test results. This 
applies to OT&E conduct and reporting only. As the SQT period is classified as 
PATS&E it should continue to be conducted with the cooperation and assistance of 
Project authorities. Similarly, to ensure test validity and impartiality, contractors 


should not be present during OT&E. 
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3. T&E Planning 
The RAN has embraced the U.S. TEMP concept as the single executive 


document for the management of T&E for major acquisitions. Although required 
early in a project, recent TEMPs have not been raised until after project funding has 
been approved. TEMPS need to be approved earlier to enable inclusion of post 
delivery PAT&E/DT&E and OT-3 in the Project Budget, and OT-4 in MHQ budget. 
The authority for DCNS to release the TEMP was based on his role as Project 
Sponsor. With HQADF now assuming this role, the DCNS function in TEMP 
development is unclear. The TEMP review process should be reviewed as part of the 
proposed OT&E system. 

A T&E Planning Group (TEPG) is currently formed prior to TEMP 
development under the leadership of the T&E manager for the project to analyse 
T&E requirements and estimate resources required. The TEPG is similar to the USN 
TECG with one significant difference. The USN TECG is led by the Director, Test, 
Evaluation and Technology Requirements (NO91) under the CNO, whereas the 
RAN TEPG is led by each individual Project Director through this T&E manager. 
Despite their best intentions, Project Directors are essentially driven by cost and 
schedule considerations, not the overall T&E adequacy of their project. To be truly 
objective, each TEPG should be chaired by the authority responsible for ensuring 
the adequacy of the RAN’s overall T&E program. The proposed RAN OT&E policy 
maker and overseer, DGNPW’s organisation, may best be suited to the chairmanship 


of each TEPG. 


4. Combined DT&E and OT&E 
With the recognition of the differences between DT&E/PAT&E and OT&E 


the RAN can now appreciate the advantages and difficulties of combined DT&E/ 
PAT&E and OT&E. By classifying SQT as PAT&E, and conducting a separate OT- 
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3 period for first of class, the conduct of combined and/or concurrent DT &E/PAT&E 


and OT&E can be addressed on the requirements of each individual trial. 


5. Foreign Weapons Evaluation 

Despite the most recent Australian Government “White Paper” on Defence 
Policy [Ref. 6:p. 70] requiring the need to be able to determine the performance in 
the Australian environment of equipment of both overseas and local origin, no DoD 
or RAN T&E policy specifically addresses the evaluation of foreign systems. The 
proposed revision of the OT&E system, and consequent DT&E revision, would 
result in a more credible and useful organisation. Policies could then be developed 
regarding OT&E and DT &E in the evaluation of foreign weapons systems. The 
current process of evaluations of foreign weapons by adhoc Project Teams could be 


modified to include greater formal involvement by T&E authorities. 


6. Land Based Test Sites 


LBTS can be used to highlight problems of operational effectiveness and 
suitability early in system development. Although LBTS are part of both recent 
major RAN combatant projects, the conduct of OT&E in these sites is planned to be 
limited to a non-interference basis with system development and integration. OT&E 
needs to be conducted during a dedicated period in the LBTS following successful 
DT&E, to gain an estimate of the potential operational effectiveness and suitability 
of the system. Again, the need for early planning to include OT&E in schedules and 


funding is highlighted. 


7. Whole Ship Testing 


The current OT&E to support an Acceptance into Naval Service (AINS) 
decision (OT-3) is little more than an extended SQT period where DT&E/PAT&E is 


conducted with some operational assessment. It is not considered to be adequate to 
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assess the first of a new ship class against the user requirement. To assess the 
operational effectiveness and suitability of a ship against the user requirement with 
the degree of confidence required for the Acceptance Board to support an AINS 
decision, a dedicated, free play, scenario based, OT-3 period is required. The 
proposed Delivery to AINS trials schedule, which is based on that currently 
proposed for HMAS COLLINS is shown in Figure 27. 
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Figure 27: Proposed Delivery to AINS Trials Schedule 





OT-3 should be conducted after the ship has completed the SQT and 
Operational Readiness Evaluation to ensure that the testing is conducted on a 
worked up and materially proven ship. To be effective the testing should be 
conducted of the overall ship as a complete weapons system. The recent USN 
innovation of “whole ship testing” provides a good example of the testing proposed. 


Whole ship testing includes engineering and technical problems combined with 
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combat system activity. [It emphasises operational realism by testing the ship as 
though in war, and is designed to provide a realistic threat in an operational 
environment, and an element of surprise to provide realistic tests to support 
completion of the OT&E objectives and resolutions of the COls. 

The OT-4 period leading to Operational Acceptance (OA) proposed in this 
OT&E model is also based on the schedule currently proposed by CTE for HMAS 
COLLINS. An example of an AINS to OA trials schedule is shown in Figure 28. 
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Figure 28: Proposed AINS to OA Trials Schedule 


It includes a long term RM&A and ILS evaluation, further testing to 
achieve statistically significant results in critical areas and the assessment of the 
operational effectiveness and operational suitability of the ship in a major exercise 


and on deployment. 
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E. RAN SHIP ACCEPTANCE 


The Acceptance Board, with its broader experience and knowledge than Trials 
Authority, is generally agreed within the RAN as being successful as a board of 
review of the planning and results of T&E for individual projects, and 
recommending PA and AINS. As an ad-hoc organisation consisting of various 
functional specialists brought together part time, however, the AB suffers from a 
number of problems. 

The AB requires guidance as to its role and functions, T&E philosophy and the 
procedures of the acceptance process. Although the AB attempts to obtain sufficient 
guidance from the T&E documentation, it has found this documentation to be 
unclear and conflicting. To enable AB consistency and development, the Board 
needs an administrator, or permanent VP. RANTAU was the administrator and 
provided guidance to the Board, with CTE now assuming this role. However, the 
AB should be independent from the T&E authority as it assesses the trials planned 
and conducted by that authority. 

With only DT&E/PAT&E and the minimal operational aspects of the previous 
OT-3 testing conducted by the RAN, the AB have tended to focus on materiel 
deficiencies. With a more comprehensive OT-3 period proposed to be conducted, 
the AB members could more effectively use their knowledge and experience in 
assessing the results of OT&E against the user requirement rather than on 
workmanship and materiel deficiencies. 

The AB should be convened for the first of class only, providing the specialist 
expertise to interpret requirements and determine the level of acceptability required 
for AINS. The AB consists of Naval Officers drawn part-time from their normal 
jobs, and so the cost appears to be minimal. The AB, however, has a high 


Opportunity cost, which is the value of the next best alternative on which the 
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members could be working instead of serving on the Board. Once the first of class 
is accepted the Board’s task is complete. The interpretations and levels of 
acceptability established by the AB then guide the trials authority in recommending 


acceptance of each of the remaining ships. 


F. RAN OT&E PROCEDURES 


1. Assessment of Operational Effectiveness 


With the proposed inclusion of a dedicated OT-3 period, the OT&E 
authority will need to develop tests for a free play, scenario based environment 
specifically designed to assess the COIs against the User Requirements. This could 
be achieved along USN lines where each COI and its related operational 
effectiveness objective is examined and the information needed to be known to 
enable each objective to be assessed is determined. The test design would have to 
take into consideration data collection methods and range considerations. The 
analysis of these tests may be different from the weapons analysis currently 
performed. Current weapons analysis focuses on individual weapon system 
performance, usually from “fire push” to “weapon destruct”, whereas whole ship 
testing in a free play environment would require a broader analysis. An increased of 
use modeling & simulation may also be required to assess some systems in a cost 
effective manner. It is beyond the scope of this thesis to investigate these 
requirements, however, close liaison between the OT&E authority and the RAN’s 
current analysis groups will be required to achieve valid testing and subsequent 


analysis. 


2. Assessment of Operational Suitability 


The current operational suitability areas now assessed are considered valid. 


These are listed in Table 23. 
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TABLE 23:SUITABILITY ISSUES FOR ASSESSMENT 


Suitability Issues 


Availability 

Reliability 

Maintainability 

Safety 

Human Factors 

Interoperability 

Compatibility 

Integrated Logistic Support including: 
Maintenance Planning 
Supply Support 
Support and Test Equipment 
Technical Data and Documentation 
Manpower and Personnel 
Training and Training Devices 
Facilities 
Packaging, Handling, Storage and Transport 
Computer Support 
Configuration Control 





The assessment of operational suitability, especially for the RAN’s current 
combatant projects, should focus on interoperability and compatibility as these 
present the highest risks with the integration of proven equipment in a unique 
combination in a ship platform. 


The assessment of ILS should continue to be conducted in two phases: 


¢ ILS Structural Review. Conducted prior to operational testing, this review 
assesses if the ILS intended to provided will support the ILS policy of the 
Project. 


¢ ILS Operational Review. During the operational testing period, the ship’s 
ILS related documentation and records are reviewed, maintainability 
demonstrations are performed and a supportability assessment is conducted. 
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RM&A data is collected and analysed during OT-3 to give an indication of 
performance prior to AINS, and then continued for 12 months during the OT4 
phase. The data should continue to be collected and analysed on an “as required”, 
vice continuous, basis by the Trials Authority until the RAN develops its routine in- 
service RM&A data collection and analysis system. The analysis of RM&A data is 
a specialist task, currently outside the experience of most uniformed personnel. To 
achieve significant and valid results, an RM&A analysis capability needs to be 


established, possibly within the Maritime Command’s weapons assessing area. 


3. OT&E Personnel, Training and Career Management 


Aneffective OT&E system is dependent on knowledgable and experienced 
personnel. OT&E is a specialised discipline, with its own philosophy and 
methodology, however, very few personnel have had any T&E experience 
knowledge prior to joining the OT&E authority. This makes training important, but 
the RAN has no training courses on OT&E, or on T&E in general. A number of 
of ficers have completed the USN Operational Test Directors Course in recent years 
which has proved beneficial to the knowledge and understanding of OT&E within 
CTE. This course should be a pre-requisite for key billets within the OT&E authority 
until the RAN develops enough knowledge and expertise to develop its own course. 
Although the USN OTD course is only of a weeks’ duration, it is understood that the 
USN is investigating a more comprehensive course. Other authorities also require 
training in OT&E. The OTE authority needs to develop acquaint courses for other 
personnel with a need to understand OT&E e.g., Project T&E managers and 
capability requirements developers. Also to increase the awareness of OT&E in the 
RAN, the OT&E authority needs to deliver presentations regularly to Project 
Directors Courses, the ILS Management and Acquisition Course (ILSMAC), 


Defence management courses, and other appropriate acquisition courses. 
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Although some personnel may develop into OT&E specialists, no career 
path in OT&E is provided in the RAN. The warfare officer functional heads within 
the OT&E authority cannot currently aspire to command of the authority as that 
position is currently a Weapons Electrical Engineer’s billet. The proposed change of 
the head of the OT&E authority to a Warfare Officer as COT&E should provide a 
career path within the authority. As the credibility of OT&E increases, the 
knowledge and experience gained by warfare officers in this area will be seen to be 
of more benefit in future warfare related positions, thus enabling a longer term career 
path. The proposed employment of COT&E under COMFLOT should also be more 
career enhancing for operations personnel than the current arrangement under 
CSO(E). 

The professionalism and knowledge of T&E personnel could be enhanced 
by membership in a related professional organisation. The International Test and 
Evaluation Association (ITEA) is one such professional organisation. 
Headquartered in the U.S. with a chapter in Australia, the association encourages the 
development and exchange of technical information in the field of test and 
evaluation. The corporate membership of a suitable professional organisation may 


be worthwhile to the RAN. 


G. OT&E MODEL SUMMARY 
The basis for this OT&E model are clear definitions of the types and phases of 


T&E. The current definitions are amended and phases of T&E are refined to better 
suit the Australian Defence environment. These refined T&E phases are 
incorporated into an amended TSTP, termed collectively as “Integrated Tests and 
Trials”, which recognises the delineation between DT&E and OT&E. No change is 
suggested to the current relationship between OT&E phases and acquisition 


milestones. 
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The current OT&E organisation structure is modified to enable effective OT&E 
to support the acquisition process. It provides for authorities responsible for the 
management, policy making and oversight, and specification of OT&E. This 
proposed reorganisation closely achieves the aim of DI&E/OT&E separation. 

The model proposes the inclusion of a dedicated OT-3 period in a free play, 
scenario based environment. The assessment of operational effectiveness is 
specifically designed to assess the COIs against the User Requirements and will 
require a different approach to analysis. The assessment of operational suitability 
focuses on interoperability and compatibility, and the assessment of ILS is achieved 
using current methods, but with improved RM&A analysis. This enhanced OT&E 
and minor modifications to the AB should provide CNS with a more comprehensive 
and thorough recommendation for AINS. 

To be effective, OT&E ideally requires current, experienced, knowledgable 
personnel with high professional credibility within their field of expertise. To 
achieve this, the model includes adequate training, good guidance documentation, 
and a career path where OT &E is seen as career progression by capable personnel. 

This OT&E model for the RAN uses principles derived from the USN system. 
By taking consideration of resource limitations, characteristics and culture of the 
RAN, it establishes an effective OT&E organisation which is important to the 


testing, evaluation and subsequent acceptance decision of the new combatants. 
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VII. CONCLUSIONS 


A. GENERAL CONCLUSION 


The USN OT&E system has, by necessity, developed into a well organised, 
well documented and effective, if complex, system. The RAN system, on the other 
hand, suffers from conflicting and confusing T&E policy documentation, a weak 
OT&E organisation structure and a general lack of OT&E knowledge and 
appreciation. The determination of OT&E required has traditionally not been 
achieved early enough in the life of a project, and so project funding has not included 
the provision for OT&E. The RAN OT&E system, although originally based on 
USN OT&E philosophy, now differs in the importance, interpretation, application 
and focus of OT&E. As a result of organisation changes, it now lacks the ability to 
effectively manage overall T&E, let alone OT&E, at a time when the operational 
risk of ship projects is increasing. To achieve efficient and effective trials and 
acceptance of the new higher risk warships currently under construction, the RAN 


OT&E system needs to be revised. 


B. SPECIFIC CONCLUSIONS 


1. OT&E in the Acquisition Process 


Australia’s Defence strategy has progressed from a position of dependence 
on allies, to a positive acceptance of both self-reliance and regional influence. To 
achieve its Defence goals, however, Australia devotes much less of its resources and 
places less priority on R&D than does the U.S. The need for local T&E is recognised 
by the Australian Government and DoD, for both indigenous and overseas systems, 


however, the importance of OT&E in particular, is not specifically addressed in any 
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Australian T&E policy document. This lack of recognition of the importance of 
OT&E is perhaps due to the RAN procuring low risk, complete ships in the past 
(e.g., FFG-7 class) with the wealth of USN DT&E and OT&E behind it. 

U.S. OT&E is viewed as being more important as a basis for a decision to 
proceed beyond Low Rate Initial Production, than on the introduction into service 
of the final production item. This is because the “big bucks” of most Defence 
acquisitions are spent during the production phase. OT&E, however, has started to 
play an important role also in the assessment of first of class systems for introduction 


into the USN Fleet. 


2. OT&E Organisation 


The RAN and Australian DoD have reorganised over the past five years to 
create a more effective and efficient user requirements organisation, and to better 
accord with Program Management and Budgeting principles. However, as a result, 
the RAN has no T&E, or OT&E policy maker and overseer, no OT&E coordinator 
within Navy Office, and the RAN’s OT&E authority is buried within the in-service 


engineering management area of the Maritime Command. 


3. OT&E Policy 


Where USN OT&E policy has its basis in the statutes approved by 
Congress, the RAN has no such formal basis and its OT&E policy is embedded in 
the more general T&E policy. The RAN has adapted the more detailed USN T&E 
definitions to suit its own requirements. During this process, however, the RAN lost 
the distinction between the types of T&E, and often confuses OT&E with PAT&E/ 
DT&E. Overall, RAN T&E documentation is not comprehensive, inconsistent and 
fails to provide adequate guidance. 

The Commander, Test and Evaluation (CTE), on the staff of the Maritime 


Commander, is independent from the development and production agencies, but not 


We 


from the end user, as in the USN. The implications of this lack of independence is 
that the end user may have his own aims and objectives which conflict with the total 
impartiality of OT&E conduct and reporting. 

Despite the most recent Australian Government “White Paper” on Defence 
Policy requiring the need to be able to determine the performance in the Australian 
environment of equipment of both overseas and local origin, no DoD or RAN T&E 
policy specifically addresses the evaluation of foreign systems. 

Although used primarily for the purpose of conducting DT &E, the USN has 
demonstrated that early OT&E on Land Based Test Sites (LBTS) can give an 
estimation of potential operational effectiveness and suitability, and hence identify 
potential operational problems early and minimise operational risk. Although LBTS 
are being developed for both the ANZAC frigate and Collins submarine projects, the 
use of these LBTS for early OT-3 was not planned in the original project schedules, 
and so is now subject to them being used on a non-interfering basis to the contractor. 

The RAN does not clearly differentiate between the types of T&E, which 
leads to DI&E and OT&E often being conducted over the same period, by the same 
test team. Although the RAN recognises, by definition, the difference in objectives 
and methodology between the types of T&E, no limitations or guidance as to the 


possible hazards of this combined testing approach are addressed. 


4. Ship Acceptance 


As an ad-hoc organisation consisting of various functional specialists 
brought together part time, the Acceptance Board (AB) suffers from a number of 
problems. It requires guidance as to its role and functions, T&E philosophy and the 
procedures of the acceptance process. Although the AB attempts to obtain sufficient 
guidance from the T&E documentation, it has found this documentation to be 


unclear and conflicting. With only DI&E/PAT&E and the minimal operational 
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aspects of the OT-3 testing conducted by the RAN, the AB has tended to focus on 
materiel deficiencies, rather than the performance of the first of class against the user 


requirement. 


5. OT&E Procedures 


The RAN recognises that the user requirement is the bench mark for 
determining the degree to which a system is effective, and also that operational 
effectiveness is best assessed by a performance demonstration by normal operating 
personnel in the normal or given environment. Assessing operational effectiveness 
has involved analyzing each Critical Operational Issue and then employing modified 
Ship Qualification Trial or Fleet Exercise Program techniques to evaluate them. The 
RAN has very little guidance on the assessment of operational effectiveness. 

The principles of operational suitability assessment are common between 
the USN and RAN. Although assessments by the RAN have become more 
subjective over recent years, the assessment of RM&A is still in its infancy. The 
RAN requires experienced personnel to design suitability tests and analyse results. 

The authority for the Deputy Chief of Naval Staff (DCNS) to release the 
Test and Evaluation Master Plan (TEMP) was based on his role as Project Sponsor. 
Now with HQADF assuming this role, the DCNS function in TEMP development is 
unclear. The TEMP approval process requires review. 

The RAN Test and Evaluation Planning Group (TEPG) for each major 
project is chaired by each individual Project Director through his T&E manager. 
Despite their best intentions, Project Directors are essentially driven by cost and 
schedule considerations, not the overall T&E adequacy of their project. To be truly 
objective, each TEPG should be chaired by the authority responsible for ensuring 


the adequacy of the RAN’s overall T&E program. 
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The early dissemination of OT&E results, before a full analysis is complete 
and the implications assessed, can lead to other authorities taking hasty action on 
incomplete information, and can prejudice the OT&E authority’s final conclusions 
and recommendations. OT&E results should indicate system deficiencies against 
user requirements rather than equipment defects, so the results need to be fully 
analysed and the implications assessed before being reported to a wider audience. 

Within the RAN, the current OT&E to support an AINS decision (OT-3) is 
little more than an extended SQT period where DT&E/PAT&E is conducted with 
some operational assessment. It is not considered to be adequate to assess the first 
of a new ship class against the user requirement. 

To be effective, OT&E requires experienced and knowledgable personnel 
with high professional credibility within their field of expertise. To achieve this 
requires selection of suitable personnel, adequate training and good guidance 
documentation, and preferably a career path where serving in the OT&E Authority 


is Seen as Career progression by such capable personnel. 
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VIII. RECOMMENDATIONS 


The OT&E model recommended for the RAN uses principles derived from the 


USN system. By taking consideration of resource limitations, and the characteristics 


and culture of the RAN, it establishes an effective OT&E organisation which is 


important to the testing, evaluation and subsequent acceptance decision of the new 


combatants. 


A. GENERAL RECOMMENDATIONS 


It is recommended that the proposed OT&E model be reviewed, evaluated and 


implemented in the RAN to achieve an effective OT&E system to support the 


introduction of operationally effective and operationally suitable new combatants 


into the RAN. 


B. 


SPECIFIC RECOMMENDATIONS 


1. OT&E Definitions 
It is recommended that: 
a. the current definitions and phases of ship T&E be amended to better suit 
the Australian Defence environment. 
b. the current TSTP be revised to show DT&E and OT&E events as 
separate categories of testing, and these trials be termed collectively as 


Integrated Tests and Trials (ITT). 


197, 


OTS&E Documentation 


It is recommended that: 


a. 


the RAN T&E DI(N) and ABR 1921 be revised to achieve 


comprehensive and consistent T&E policy guidance. 


OT&E in the Acquisition Process 


It is recommended that: 


a. OT&E be conducted, where possible, prior to the production or 
purchase decision in the acquisition process to minimise risk. 

b. OT-3 be separated from the DT&E /PAT&E post delivery trials. 

c. OT-3 be conducted as a dedicated, free play, scenario based period after 
the completion of SQT and ORE to ensure testing is conducted on a 
worked up and materially proven ship. 

d. OT-3 of the overall ship be conducted as a complete weapons system. 

OT&E Organisation 


It is recommended that: 


a. 


the appointment of a DoD OT&E policy maker and overseer within 
HQADF be investigated. 

DGNPW be appointed as the OT&E policy maker and overseer within 
the RAN. 

a close working relationship on T&E matters be required between 
DGFD(Sea), DGNPW and the OT&E authority. 

DGNER be appointed as the Materiel Division T&E agency to assist 
Project Directors in complying with policies, incorporate lessons 
learned from previous projects, review the T&E aspects of projects at 


major milestones, and advise projects how to prepare for OT&E. 
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e. the current CTE organisation be disestablished. 

f. an “In-service Trials” team be established under CSO(E) to conduct 
CTE’s current DT&E/PAT&E responsibilities, and those already the 
responsibility of CSO(E). 

g. Commander, OT&E (COT&E) be established as the RAN’s OT&E 
authority under COMFLOT, to manage OT&E and the post delivery 
trials period for all ships from the MHQ perspective. 

OT&E Policy 


It is recommended that: 


a. 


go 


policy be developed to enable the performance in the Australian 
environment of foreign equipment to be evaluated efficiently, 
effectively and consistently. 

the TEMP approval process be reviewed to ensure appropriate 
authorities are consulted and approve the TEMP. 

the TEPG be chaired by DGNPW as the authority appointed to ensure 
the adequacy of the RAN’s overall T&E program. 

OT&E not be influenced by the Project or DAA, and that Project or 
DAA personnel not be present during testing. 

contractors not be present during OT&E. 

TEMPs for non-development projects be approved before production 
approval to enable inclusion of post delivery PAT&E/DT&E and OT-3 
in the Project Budget, and OT-4 in MHQ budget. 

OT&E be conducted during a dedicated period in the LBTS following 
successful DT&E, to gain an estimate of the potential operational 


effectiveness and suitability of the system. 
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6. Ship Acceptance 


It is recommended that: 


a. 


b. 


the Acceptance Board remain as a board of review. 


the Acceptance Board be convened for the first of class only. 


OTS&E Procedures 


It is recommended that: 


a. 


ga 


The OT&E authority develop tests for a free play, scenario based 
environment specifically designed to assess the COIs against the user 
requirement. 

analysis requirements of these tests be assessed, as they may be 
different from the weapons analysis currently performed. 

experienced personnel assist in designing suitability tests and analysing 
results. 

interoperability and compatibility be a major focus during the 
assessment of operational suitability, especially for the RAN’s current 
combatant projects. 

an improved RM&A analysis capability be established. 

RM&A data continue to be collected and analysed on an ‘as required’ 
basis until the RAN develops its routine in-service RM&A data 
collection and analysis system 

OT&E reports be made solely by the OT&E authority. 

OT&E phase reports be issued only, with no progress or interim reports, 
to allow full analysis and evaluation of the test results. 

the USN Operational Test Director type course be a prerequisite for key 
billets within the OT&E authority until the RAN develops enough 


knowledge and expertise to develop its own course. 


the OT&E authority develop acquaint courses for other personnel with 
a need to understand OT&E. 

the OT&E authority deliver presentations regularly to Project Directors 
Courses, the ILS Management and Acquisition Course (ILSMAC), 
Defence management courses, and other appropriate acquisition 
courses, so to increase the awareness of OT&E in the RAN. 

a Warfare Officer be appointed to head the OT&E authority. 

. establishing corporate membership of a suitable professional T&E 


organisation be investigated. 


APPENDIX A: USN OT&E COURSE SYLLABUS 


The Commander, Operational Test and Evaluation Force (COMOPTEVFOR) 
conducts a four day Operational Test Directors (OTD) course designed to acquaint 
prospective Operational Test Directors and other interested personnel with the 
fundamentals of OT&E philosophy, terminology, policy and procedures. This 


course covers the following subject areas: 


e¢ Acquisition Program Overview 

¢ Test and Evaluation Master Plan (TEMP) 
¢ Operational Effectiveness Overview 

¢ Operational Suitability Overview 

¢ Logistic Supportability 

¢ Fleet Research and Development T&E Support 
¢ Operational Security 

e Managing Money 

e Weapons Systems Survivability 

¢ Threat Support 

¢ Test Planning 

e Software Intensive Systems 

¢ Modeling, Simulation and Analysis 

¢ Conducting the Tests 

¢ Lessons Learned 

e Standards of Conduct 

e Evaluation Reports 

¢ Director, OT&E Perspectives 

¢ Chief of Naval Operations Perspectives 


¢ Program Manager Perspectives 


LOZ 


¢ Relationship with General Accounting Office 
¢ Test Resources 


¢ Program Formulation 
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ABBREVIATION 


A 

AAW 

AB 

ABR 
ACDEV-N 
ACMAT-N 
ADF 


AINS 
AMP 
ASC 
ASUW 
ASW 


CAP 

CEP 
CEPMAN 1 
CIWS 

CNO 

CNS 

COEA 


APPENDIX B: ACRONYM LIST 


FULL TITLE 


Antiair Warfare 

Acceptance Board (RAN) 

Australian Book of Reference (RAN) 
Assistant Chief of Naval Staff (Development) 
Assistant Chief of Naval Staff (Materiel) 
Australian Defence Force (RAN) 
Australian Frigate (RAN) 

Acceptance into Naval Service (RAN) 
Acceptance Management Plan (RAN) 
Agreed Ship Characteristics (RAN) 
Antisurface Warfare 


Antisubmarine Warfare 


Combat Air Patrol 

Chief of Capital Procurement (RAN) 

Capital Equipment Procurement Manual (RAN) 
Close-in Weapons System 

Chief of Naval Operations (USN) 

Chief of Naval Staff (RAN) 


Cost and Operational Effectiveness Analysis 


COI 

COMFLOT 
COMNAVSEA 
COMOPTEVFOR 


(CAO) 
CSE 
CSO(E) 
CSO(O) 
CSSQT 
CSE 
CFE 

D 

DAA 
DCNS 
DDG 
DECP 
DGFD(SEA) 
DGNPW 
DGNW 


DI(N) 
DLSIE 


DNUR 
DNW 


Critical Operational Issue 

Commodore Flotillas (RAN) 

Commander, Naval Sea Systems Command (USN) 
Commander, Operational Test and Evaluation 
Force (USN) 

Capital Procurement Organisation (RAN) 
Combat System Evaluation (RAN) 

Chief Staff Officer (Engineering) (RAN) 

Chief Staff Officer (Operations) (RAN) 
Combat System Ship Qualification Trial (RAN) 
Combat System Trainer (RAN) 


Commander Test and Evaluation (RAN) 


Design Approval Authority (RAN) 

Deputy Chief of Naval Staff (RAN) 

Guided Missile Destroyer 

Defence Force Capability Proposal (RAN) 
Director General Force Development (Sea) (RAN) 
Director General Naval Policy and Warfare (RAN) 
Director General Naval Warfare (RAN) 

Defence Instruction (Navy) (RAN) 

Defence Logistics Studies Information Exchange 
(USN) 

Director of Naval User Requirements (RAN) 
Director of Naval Warfare (RAN) 
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DoD 
DOT&E 
DSMC 
DT&E 


ECM 
ESM 
EW 

F 

ECE 
FFG 
PLEETEX 
FOT&E 
leeD) 

G 

GAO 
GDP 


HATS 
HMAS 
HQADF 


1S 
IES 
IOT&E 


Department of Defence 
Director of Operational Test and Evaluation (USN) 
Defence Systems Management College (USN) 


Development Test and Evaluation 


Electronic Countermeasures 
Electronic Surveillance Measures 


Electronic Warfare 


Final Contract Trials (USN) 

Guided Missile Frigate 

Fleet Exercise (USN) 

Follow-on Operational Test and Evaluation (USN) 
Full Scale Development (USN) 


General Accounting Office (USN) 


Gross Domestic Product 


Harbour Acceptance Trials (RAN) 
Her Majesty’s Australian Ship 


Headquarters, Australian Defence Force (RAN) 


Integrated Logistic Support 
Integrated Logistic Support Plan 
Initial Operational Test and Evaluation (USN) 


ITP 
ITT 


EBS 
LRIP 


MHQ 
MNS 
MOE 

N 
NAVSEA 
NCP 
NWAC 

O 

OA 

OCF. 
OPD 
OPEVAL 
OPTEVFOR 
ORD 
ORE 
OSD 
OTA 
OTD 
OT&E 


Integrated Test Package (RAN) 
Inspections, Tests and Trials (RAN) 


Land Based Test Site 
Low Rate Initial Production (USN) 


Maritime Headquarters (RAN) 
Mission Need Statement (USN) 


Measures of Effectiveness 


Naval Sea Systems Command (USN) 
Naval Capability Proposal (RAN) 
Naval Weapons Analysis Centre (USN) 


Operational Acceptance (RAN) 

Operational Concept Paper (RAN) 

Operational Performance Demonstration (RAN) 
Operational Evaluation 

Operational Test and Evaluation Force (USN) 
Operational Requirements Document (USN) 
Operational Readiness Evaluation (RAN) 
Office of the Secretary of Defence (USN) 
Operational Test and Evaluation Agency (USN) 
Operational Test Director (USN) 


Operational Test and Evaluation 
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PA 

PAT 
PAT&E 
PD 
PDA 
PDT &T 
PM 
PMBS 


PREINSURV 
PSA 

R 

R&D 


RANTAU 
REFTRA 
RM&A 
ROC 

S 

SATS 
SECDEF 
SQT 

Suey 

SOE 


Preliminary Acceptance (RAN) 

Production Acceptance Test (RAN) 
Production Acceptance Test and Evaluation 
Project Director (RAN) 

Post Delivery Availability 

Post Delivery Tests and Trials (RAN) 
Program Manager (USN) 

Program Management and Budgeting System 
(RAN) 

President, Board of Inspection and Survey (USN) 
Post Shakedown Availability (USN) 


Research and Development 

Royal Australian Navy 

RAN Trials and Assessing Unit (RAN) 
Refresher Training (USN) 

Reliability, Maintainability and Availability 
Required Operational Characteristics (RAN) 


Sea Acceptance Trials (RAN) 
Secretary of Defence 

Ship Qualification Trial (RAN) 
Strike Warfare (USN) 
Schedule of Events (USN) 
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SUPSHIPS 
i 

T&E 
PECG 
JUeivie 
HEeG 
TER 

TSiB 


USN 
USS 


WSRL 


Supervisor of Shipbuilding (USN) 


Test and Evaluation 

Test and Evaluation Coordination Group (USN) 
Test and Evaluation Master Plan 

Test and Evaluation Planning Group (RAN) 
Top Level Requirement (USN) 


Total Ship Test Program 


United States Navy 
United States Ship 


Weapons Systems Research Laboratory (RAN) 
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